Elevate your Metadata platform to become an immensely data-driven organization

The paradigm shift from Passive to Active Metadata management is critical for all data-enabled organizations to be successful.

Jul 15, 2023

Do organizations need a Metadata Management Platform? Are the existing Metadata Management Platforms incapable of fulfilling the growing needs of enterprises? How to get started with enabling Metadata Management? Let’s understand the insights that can help with the right implementation of Metadata management needs.

What is Metadata?

Since 1990, we know that metadata in its basic form means 'data about data'. Insights about the critical data assets in an enterprise landscape, capturing their attributes, and their association represented as lineage. The advantage of metadata is it does not store the actual rows or data values like the name of customers; however, metadata focuses on the data structure and its information holding these data values like the table name and its column name.

The metadata model is built to capture the data assets & enable a view to display it as a metadata catalog (generally known as Data Catalog). With a Data Catalog, the organization is sharing the metadata information about these data assets to the end-users and at the same time protecting the sensitive and confidential data values that may not be authorized for broader organization view. This spreads a great understanding and awareness of enterprise data in the organization and also stays in compliance with data security, governance, and privacy guidelines, leading toward “Data Literacy”1

Data catalog in itself is a topic for more in-depth discussion, so I will be covering that in upcoming articles; however, the focus of this article is on the Metadata management process.

graphical user interface — Photo by Choong Deng Xiang on Unsplash

Metadata management tools, that are available for implementation of metadata solutions, largely manage the following aspects of the enterprise data landscape.

Metadata model- that defines a model to capture key data assets, and their attributes to identify the ownership and associations. Additional user-defined documentation/ details on the critical attributes like PII (Personally Identifiable Information) or other sensitive data within the data objects to ensure proper consumption of such data. The entities in the metadata model may have references to other data models2 (conceptual, logical, or physical models) in the organizations. Such lineage, i.e. their source of origination, and their targets of consumption helps expand the model organically grow into an enterprise metadata model.
Metadata processes- the lifecycle of data assets defined in the metadata model, their statuses, for example, active or archived, and the history information like the user who created, updated, or deleted (if allowed) these data assets and when. The information about what are the archival, and data purging processes to safely terminate these data assets from the enterprise. Any linkage to critical, sensitive, or confidential data assets to analyze impact, risk, or cascade termination strategies. Metadata management tools that capture such information about the metadata assets are significant for organizations to understand how to consume and treat data assets respectfully.
Business glossaries- defining policies, terms, and a general understanding of what each attribute means, in the enterprise context, along with the Thesauri information. Business glossaries are useful knowledge bases to create common semantics in organizations so that each data user understands and uses the common terms defined in business glossaries to collaborate.
Taxonomies- the business terms can then be represented in a hierarchical structure using taxonomies or classification of the data assets defined in the metadata model. They are simple representations like classes or groups. Taxonomies help understand the soft association of business terms and their classification.
Ontologies- Each of the terms or taxonomies can further link in a relational model using ontologies. Each of the hierarchical relationships can have rules or additional classification information. Ontologies are the best representation of knowledge graphs.
Security & Compliance- addition of compliance information like GDPR3 or CCPA4 guidelines and mapping the critical data elements that are attached to these compliance requirements. This helps the data user to quickly react to manage risk exposure and avoid being non-compliant.

Common Type of Metadata

1. Business Metadata capturing the business capabilities like defined processes to manage the lifecycle of data, organizational aspects like roles, agreements, owners, and semantic aspects like business rules or definitions.

2. Operational Metadata capturing key data quality aspects like metrics, scores, dimensions, and KPIs along with any data monitoring rules, alerts & notification information for reporting the health and quality of data assets.

3. Technical Metadata capturing the information about the system or application of the data asset, its connectivity details, protocols supported, the technology ownership, point-of-contact, operating hours or downtime schedules, and platform information.

4. User-defined Metadata capturing additional information about data assets that can be useful to determine the usage and ownership.

5. Social Metadata capturing comments, tags, ratings, likes, notes, and labels for collaboration purposes.

Advantages of Metadata Management Platform

Enterprise Metadata- Organizations growing with rapid scale and volumes of data requires a metadata management platform to help with managing all the enterprise data assets in one location such that it keeps growing with the organization's needs.
Centralized governance- Well-established data assets and enablement of key capabilities like the life-cycle of data assets, tagging of critical data elements, and lineage information, helps the responsible data governance team to make data decisions confidently.
Trusted Data- Value-added knowledge bank, updated documentation about the data assets help end-user to spend less time finding the right information and more time analyzing and driving value with the data.
Security and Compliance- Organizations requiring to perform periodic regulatory compliance audits can drive such requirements with capabilities that bring reusability and optimizations in the process with end-to-end visibility and tracking.

Drawbacks of today’s Metadata Management Platform

Metadata management processes have now become passive and insufficient as the organizations have started advancing in analytics, have achieved capabilities to expand data science and machine learning models, and simply acquire new sources through mergers & acquisitions.
Moreover, with the rapid pace at which alternate technologies are available like NoSQL, Vector, or Graph databases, metadata is more dynamic and diverse as the data assets started growing on-premise and in the cloud. In short, it is spreading across the entire ecosystem, instead of traditional relational databases and data models. It now exists in more technologies than just traditional databases alone, for example, applications, data-integration tools, MDM tools, Cloud services, infrastructures, etc.
The stale nature of metadata elements and their lineage that does not capture the holistic enterprise view metadata is not sufficient for data-driven business outcomes.
Most of the traditional metadata management tools have a narrow/ siloed approach toward cataloging data. This has limited sharing capabilities that limit the metadata context to local or domain metadata instead of enterprise metadata.

Hence, there is a need for metadata management processes to go beyond the stale documentation system that is not widely sharable across the enterprise ecosystem.

The Paradigm Shift from Passive Metadata Architecture to Active Metadata Architecture. - Image By: Suhas Dhekane

The Paradigm Shift from Passive to Active Metadata Management

Active Metadata elevates the context of metadata by harvesting metadata across the enterprise landscape. This is done by collecting more than just technical metadata and also includes operational, business, and social aspects of metadata from the providers and consumers. Moreover, it also harvests this information from the layer that provides accessibility to data and performs data transformation.
With an enterprise-wide view of metadata that extends a passive metadata model, the notion of active metadata can help eliminate data silos. Data Silos emerge when groups work within their isolated boundaries or have an awareness of other data assets. This is a major cause of data problems in organizations.
The set of capabilities that active metadata unleashes helps with the continuous growth of the metadata model as new assets are discovered or acquired. This helps to improve data analysis to derive an understanding of data that was previously unexplored. Furthermore, it allows getting prescriptive recommendations based on execution results, and it can report on continuous health score analysis, KPIs, or outcomes achieved.
Active Metadata is at the transformational phase of the evolution in data-enabled technologies.

The demand for active metadata is to assure augmented data management capabilities to automate & optimize data all the time.

If you are new and have not read my previous article on Master Data Management (MDM)5, I will highly recommend reading that to understand what is augmented MDM and the use cases. Click this button to read that article.

Link to MDM Trends

Challenges to Enable Active Metadata

The challenges may vary based on the overall organization’s maturity towards its metadata management journey. However, assuming that organizations have a metadata management platform that aligns with the passiveness notion mentioned above, such organizations may see the below challenges-

Scattered Metadata- The metadata is scattered everywhere in the organization. It is difficult to understand the way metadata information is stored within the local or domain boundaries with a limited understanding of that domain-specific knowledge outside those boundaries.

Interoperability- There is a lack of common metadata standards, which makes metadata sharing and interoperability a major challenge across multiple metadata management solutions in the market. Such issues of interoperability also arise because of silos in organizations.

Accessibility- Data management platforms such as databases, data integration, data quality, and data governance tools have consistently increased their capabilities relative to accessing and managing data. Metadata capabilities embedded in most data management solutions cannot provide a self-service portal to business users to access metadata for analysis of all types of metadata categories from any platform.

Key Objectives to Overcome the Challenges-

A good starting point for overcoming such challenges is discussed in this section. This is not the complete set of objectives, but they are the essential elements to start building maturity of the existing metadata management platform towards enabling active metadata notion.

Unified Model- It is key to start with the definition of the enterprise-wide view of the metadata model. The organizations can extend an existing passive metadata model and can organically grow into an enterprise model across all tools, technologies, and practices. Please note, the reference here is on the conceptual model instead of a logical or physical model. The focus is more on identifying the following-

Cross-platform data attributes are critical for organizations and hence the need to model them at a central metadata location that will be available for all end-users in the organization. The less critical data attributes can stay in the current state architecture and be shared or accessed through the local metadata repository when needed. Metadata sharing is the key and recommended approach for active metadata enablement. Various architecture patterns can further provide deep insights into implementing these capabilities; however, these architecture patterns are a future scope of discussion in upcoming articles.
Once the enterprise metadata model starts shaping up, the focus can be to associate the lineages. There are two types of lineages - horizontal and vertical lineage. The source system or data provider and target systems or data consumer association represent a horizontal lineage, whereas the association of one data asset to another defines the vertical lineage. Both types of lineages can be well represented by graph-optimized databases or knowledge graphs6.
Other key focus areas are to define the processes that govern the life-cycle of data and their dependencies to continuously manage both the lineage. There are ways to automate this so that changes to data assets are system-driven, with a workflow to approve these changes as part of the continuous metadata harvesting step.
Lastly, to complete this exercise, it is important to define the ownership of schemas, structures, or models related to the data to understand and model the end-to-end governance processes for these structures. Every data team takes up the federated responsibility for their data domain to drive the changes within the domain through a roadmap. It helps with clarity, and separation of duties and eliminates the data silos leading toward treating each data domain as a ‘Data Product’ to build advanced data practices or “Data Mesh” architectures.

Metadata Integration- This is also key for ensuring the interoperability of a metadata model that can be consumed enterprise-wide as a trusted source of metadata information. The well-known integration patterns apply to integrating metadata information. Metadata can be made available through a push mechanism via messaging or streaming. It can also be pulled on-demand by invoking REST APIs hosted by the metadata layer. This layer hosts the key information that must be made available to the target system; information in the form of business or user-defined metadata, technical metadata that is tagged for sharing, business glossaries or business terms, and social aspects like rating of metadata source or searching metadata based on available tags.

Self-service Metadata- The above two points are significant from the design and implementation of the metadata management platform; however, the focus on making this platform available to end-users for metadata consumption is key to the usage and adoption of metadata.

A large percentage of metadata management tools or existing metadata management implementations fail to enable self-service metadata capabilities. Without the ease of use of the metadata platform, the organizations struggle with adoption and hence the ROI on such investments cannot be rationalized.

There are a few considerations to ensure the metadata management layer achieves the maximum adoption in the enterprise. Enablement of these capabilities is key, but the decision of when or how to enable them may depend on the maturity of the organization's practices. The following are additional capabilities considerations.

The metadata layer enabled with active metadata capabilities is a unified view of enterprise-wide metadata. Data analysts, Data Stewards, and various practitioners of Data across the enterprise will be leveraging this tool day in/day out. Making this a self-service portal where it is easier to achieve the following-
- look-up, search data catalogs, business glossaries, terms,
- interact with data through social aspects like tags, ratings, and comments for collaborations,
- making significant decisions related to the security and compliance of data Identifying data risks, impact, and mitigation decisions.
The other key aspect is the self-service provisioning of data. The metadata layer is enabled with governance processes. This is a great source to identify new needs and provision new data views, and catalogs as needed to solve new business requirements and publish them for enterprise consumption.

Enabling these self-service capabilities will not only bring all the data players together to collaborate on the unified platform but also implement a common understanding and semantics that will open opportunities for advanced data practices like “DataOps” and ensure the notion of democratizing data.

Key Recommendations to Enable Active Metadata-

As organizations continue to invest in the data management platform, it is key to ensure that the following capabilities are a continuous focus of the platform roadmap:

Metadata Sharing- Adjacent data management tools must be able to share internal metadata information with the metadata management tool for broader and end-to-end metadata analysis and orchestration. The recommendation is to choose data management tools that allow metadata sharing that can be easily integrated with the metadata platform.
Runtime Metadata- Organizations invest in UX analytics for abilities to capture the user & data interaction and overall data usage patterns. Such patterns are required for continuous analysis of what data assets are preferred or others and why. This is a well-known concept of ‘data affinity’ in data-mining7, which can be achieved through automation and UX analysis which will further exploit the metadata value.
Observability- The key capability for metadata management tools is to provide native support or have easy integration with data observability tools to create prescriptive recommendations and insights on operational aspects of data.
Metadata Import/Export- Along with Metadata sharing, there can be a need to export and import metadata across the enterprise. The metadata management tool should be able to gather process and optimize such metadata without having to perform heavy transformations. Such common features help with interoperability and achieve a standard metadata approach.
Metadata Analytics and Governance- Changes to Metadata and its objects are inevitable. The changes can be determined in the metadata tool or implicitly requested through an adjacent data management tool. Either way, there should be an automated trigger to author, review and approve these changes through a metadata governance/ workflow model. Analyzing such changes, their impact, and risk would be a natural next step to understand and mitigate downstream impacts. With the recommendation of building data teams having federated responsibility of defining their ‘Data Product’ roadmap, the governance of metadata will be achieved.

Conclusion:

Metadata management is a significant functionality in practically all data-enabling technologies and metadata analytics, augmented and automated design practices, and even deployment of data management platforms will continue to be a critical aspect of data-driven businesses.
Gartner's analysis8 suggests that this is a faster-growing software market with the highest growth rate & adoption in the past 2 years of 21.6% reaching nearly $2B.
Metadata management tools with passive metadata management capabilities will continue to drive the implementations in data-enabled organizations that are either starting with a metadata journey or on a lower maturity curve with metadata.
However, the need for active metadata processes and techniques will continue to evolve. The discussed capabilities for active metadata management will be natural next steps of implementations in already established metadata management tools.
The active metadata management notion is here to stay and will accelerate in the upcoming years through the adoption of “Data Fabric” Architecture implementation.