In all data architectures, be it a data mesh, data fabric, data warehouse, or data lakehouse architecture, metadata is indispensable. This is especially true for business users who develop their own dashboards and reports and create data science models. To offer frictionless access to metadata, it is important that, when designing a data architecture, time is reserved for designing a full-fledged metadata architecture. Metadata should not be an afterthought or treated like a neglected stepchild. It is the key to a successful data architecture.

Target Audience: Data Architects, Enterprise Architects, Solutions Architects, IT Architects, Data Warehouse Designers, Analysts, Chief Data Officers, Technology Planners, IT Consultants, IT Strategists
Prerequisites: Some understanding of data architectures
Level: Advanced

Extended Abstract:
In the past, metadata was mainly kept for IT specialists. They needed detailed descriptions of what all the files, tables, and columns meant. That need still exists, but new users are interested in metadata.

Today, business users, especially those who develop their own dashboards and create data science models, need access to metadata. They need to be able to see descriptions, categorizations, lineage, and so on. They need to know exactly what they are looking at. They must be able to search metadata to find the right data to develop a report or data science model. They also need metadata to clarify how to interpret the data. So, we need to analyze in detail what types of metadata business users need.

It is also important to study how they want to access the metadata. For example, for a business user working with a dashboard full of financial data, descriptions of what all those values mean should be readily available. And they may want to see that metadata automatically when they hover their mouses over a specific value for two seconds. They may not want to start a separate system to get those descriptions, because it would be inconvenient and time consuming. It is important to understand how users want to use metadata. Business users may want to add their own metadata descriptions in the form of annotations. Again, to design the right metadata architecture, this would be an important requirement.

Currently, metadata must be properly managed for reasons of data privacy, security, auditability, and governability, which may require robust forms of version control. 

Metadata must also be entered and extracted from other systems. With respect to extraction, an ETL-like solution may be required to extract metadata from source systems and schedule these extraction processes periodically, as metadata can change. Keep in mind that ETL for metadata does not simply extract metadata from some structured data source. It can mean extracting metadata descriptions from ETL programs, database stored procedures, and the semantic layers of reporting tools. The source for metadata-ETL is not always a database, but sometimes also code and specifications.

All of these requirements do not magically appear; they must be analyzed, designed, and developed. In all data architectures, be it a data mesh, data fabric, data warehouse, or data lakehouse architecture, metadata is indispensable, especially for modern business users who develop their own dashboards and reports. To offer frictionless access to metadata, it is important that, when designing a data architecture, time is reserved for designing a full-fledged metadata architecture. Metadata should not be an afterthought or treated like a neglected stepchild. Metadata is the key to a successful data architecture.