Share this article
What is Metadata? #
Metadata is defined as the information that describes and explains data. It provides context with details such as the source, type, owner, and relationships to other data sets. So, it can help you understand the relevance of a particular data set and guide you on how to use it. In a nutshell: Metadata is a cornerstone of a modern enterprise data stack.
See How Atlan Streamlines Metadata Management – Start Tour
As Ashish Thusoo and Joydeep Sen Sarma put it in their book titled Creating a Data-Driven Enterprise with DataOps, metadata is data about data.
Here’s how the University Libraries from the University of North Carolina lend more context to the definition of metadata:
Metadata describes a data set (by providing answers to questions such as):
- How it was collected?
- When was it collected?
- What assumptions were made in the data collection methodology?
- What is the geographic scope?
- Are there multiple files? If yes, how do they relate to one another?
- What are the definitions of individual variables and, if applicable, what were the possible answers?
- What was the calibration of any equipment used in data collection? And the version of software used for analysis?
Already sold on the potential of metadata? Learn how to bring modern metadata into your modern data stack. Download Free Primer.
Table of Contents #
- What is metadata?
- Metadata examples
- Types of metadata
- Why is metadata important?
- How does metadata add context to data and help data teams?
- Metadata use cases
- Metadata as the foundational block for DataOps, data mesh, and modern data governance
- How to manage metadata?
- Conclusion
- Related reads
Examples of metadata #
Let’s look at metadata examples of two common file formats - an image and a spreadsheet.
An Image file #
Examples of metadata - an image file. Image by Atlan.
For example, the above image file includes the below metadata:
- The make of the camera
- Lenses used
- Time at which the picture was taken
- Focal length
- GPS coordinates of the location
- Image resolution
- Color profiles
This metadata offers technical insights about the image, which is useful during image processing. It also helps you easily search for the images you need.
A Spreadsheet file #
let’s consider another example — a spreadsheet.
Examples of metadata - a spreadsheet file. Image by Atlan.
The table row and column headers in a spreadsheet are examples of metadata as they offer context to the data.
Other metadata examples include:
- Number of rows and columns
- Source descriptions and relationships
- Validation rules for a data asset
- Data types
- Column statistics — missing values, min-max values, and histogram distribution
- Data owner, creation date, comments, and information on the latest updates
Data vs. metadata #
Data is raw information, whereas metadata is the context of that information.
So, data can include any information — measurements, customer names, regional sales volume, supplier names, and more.
However, without metadata, there’s no way of knowing whether a particular piece of information is a measurement, sales amount, or the total sales from a certain region. So, metadata is the contextual information that helps you understand raw data.
Metadata and data of a computer file. Image by Atlan.
Read more → Data vs. metadata
Before we move forward, it’s essential to keep in mind that the language and terms we use will reflect how data teams operate in businesses.
What are the types of metadata? #
Metadata can be classified into 6 types:
- Technical: This includes technical metadata such as row or column count, data type, schema, etc.
- Governance: This includes governance terms, data classification, ownership information, etc.
- Operational: This includes information on the flow of data such as dependencies, code, and runtime
- Collaboration: This includes data-related comments, discussions, and issues
- Quality: This includes quality metrics and measures, such as dataset status, freshness, tests run, and their statuses
- Usage: This includes information on how much a dataset is used, such as view count, popularity, top users, and more
6 types of metadata. Image by Atlan.
While managing metadata, it’s helpful to look at it as active and passive metadata.
Active metadata vs. passive metadata #
Passive metadata is the technical metadata collected and managed via manual processes.
Active metadata is data that describes data, as well as any actions taken on it. Unlike passive metadata, active metadata is an always-on, intelligent, action-oriented, API-driven system that is open by default.
A Forbes article puts it this way:
Active metadata implies that both the metadata itself and the management are active.
Meanwhile, Atlan’s co-founder Prukalpa Sankar emphasizes its significance with this example:
“Active metadata sends metadata back into every tool in the data stack, giving the humans of data context wherever and whenever they need it — inside the BI tool as they wonder what a metric actually means, inside Slack when someone sends the link to a data asset, inside the query editor as they try to find the right column, and inside Jira as they create tickets for data engineers or analysts.”
To gain a deeper understanding of active metadata and its management, check out this article on active metadata management and its importance in today’s data stack. This write-up will provide insight into the significance of active metadata management and how it can benefit the modern data stack.
The Ultimate Guide to Data Mesh - Learn all about scoping, planning, and building a data mesh 👉 Download now
Why is metadata important? #
The right context for data is essential for understanding and putting it to use. Metadata helps you make data discoverable, accessible, trustworthy, and valuable.
A guide on metadata from University of North Carolina states that without metadata, a data set is incomprehensible. This highlights the importance of metadata in understanding data sets.
Meanwhile, the author of Understanding Metadata Jenn Riley believes that metadata is the “key to the functionality of the systems holding the content, enabling users to find items of interest, record essential information about them, and share that information with others.”
How does metadata add context to data and help data teams? #
Metadata ensures that data is:
- Discoverable
- Trustworthy
- Relevant
- Accessible
- Secure
- Interoperable
1. Discoverable #
Metadata makes data easier to find using a text-based search. Each search result will provide enough context to explain what it is about. Metadata also helps to sort and filter through the search results, thus significantly reducing the time spent in data search and discovery.
2. Trustworthy #
Metadata helps to track the lineage of each asset throughout its lifecycle. Documenting changes to an asset over time makes it easier to verify, validate, and use. For example, it is possible to check for issues whenever changes are made to data and notify the relevant downstream data users.
3. Relevant #
It is essential to know whether the data stored is up-to-date and useful for the data teams. For instance, if no one has accessed a data set in over 120 days, then it is either stale or not discoverable. If it is stale, it can be purged completely.
4. Accessible #
Once the data needed is found, metadata provides details on how to access it — authentication and authorization methods.
5. Secure #
As mentioned earlier, metadata can be used to trace who accessed or modified a specific data set. It also helps to keep track of the various versions of data and classify data to set up data access controls. This, in turn, ensures the security and integrity of data.
6. Interoperable #
Proper metadata management ensures that the entire tech stack — different systems, platforms, and tools — speaks a common language, thus making the systems interoperable.
What are some use cases of metadata? #
Metadata plays a significant role in everything from data discovery to lineage and governance. So, let’s look at three prominent metadata use cases:
- Speeding up root cause analysis
- Managing security classifications
- Optimizing data stack spending
1. Speeding up root cause analysis #
It’s not uncommon for data teams to spend several hours trying to find out what went wrong while running a query or generating a report. Automated column-level lineage helps in depicting the flow of data visually — right from its origins to the transformations it underwent. Setting up this form of lineage mapping requires tracking all kinds of metadata, in addition to the queries run and reports set up.
2. Managing security classifications #
Every organization has several classification levels for data. For instance, regulations like the EU GDPR require you to segregate and encrypt PII information. You also have data that are used by various teams, such as HR or Finance. So, it’s important to automatically classify data and ensure that only people with the right credentials can access it and that’s where metadata can help.
3. Optimizing data stack spending #
Metadata can help you analyze the most and least used tables, tables being used by BI tools, or queries that take too long to run. This information can be useful in sprucing up your data warehouse and optimizing its usage.
If you want to check out some use cases of metadata, here’s a video that lists 20 metadata use cases in 20 minutes.
Lastly, metadata plays a substantial role in helping modern data teams enforce data governance policies, track data lineage, monitor data quality, and more. Let’s see how.
Metadata as the foundational block for DataOps, data mesh, and modern data governance #
Metadata and DataOps #
DataOps is a collective effort by the humans of data to improve communication, automation, and data flow integration within an organization.
The ultimate goal of any data team is to find, process, and interpret the data they need to quickly gain vital insights that help them do their jobs. That’s where a DataOps framework can help.
According to Gartner, data and analytics leaders can use DataOps to enable developing insights quickly while following data disciplines and procedures.
For the framework to work, you must understand how data flows and gets consumed within your organization, i.e., effective metadata management.
Read more → What is DataOps?
Metadata and the data mesh #
The Data Mesh is a decentralized data architecture proposed by Zhamak Dehghani (ex-director of Emerging Technologies for ThoughtWorks in North America), where data is treated as a product and managed by dedicated data product owners.
The goal of this architecture is to ensure there is data ownership and accountability to eliminate data accessibility, discovery, and trust issues. To ensure this in practice, metadata must be discoverable, accessible, and trustworthy.
Read more → What is data mesh and what’s the role metadata plays in it?
Metadata and modern data governance #
Modern data governance is a personalized, community-centered approach toward data enablement. The goal is to help you understand what data you have, where it came from, how it has changed, and how to use it.
Thoroughly documenting metadata on an asset’s origins, data type, transformations, and more is the first step to setting up a successful data governance framework.
Read more → Data governance and effective metadata management
How to manage metadata? #
Metadata management is the practice of cleaning, classifying, and organizing data to ensure its accuracy, integrity, consistency, and usability.
It’s the key to cataloging data assets in your organization and gives you the ability to trust your metadata. Using metadata management tools, you can set up a metadata repository and enable metadata discovery, lineage mapping, and governance.
To get started with metadata management, you must develop a metadata strategy leveraging all types of metadata and set up an environment where metadata is easy to discover, understand, and use.
Read more → What is metadata management?
People also asked questions #
What is metadata lineage? #
Data lineage is information about the origin and evolution of data. It helps understand where data has come from and how it has evolved through its lifecycle by tracking all metadata, SQL queries, BI reports, and more.
Read more → Data lineage 101
What is a metadata catalog? #
A metadata catalog acts as a central, searchable repository for all metadata and data. This can include attributes such as data source, origin, owner, definition, usage, and more.
Read more → What is a metadata catalog?
Conclusion on Metadata #
Robust metadata management is the key for data-driven teams to discover, understand, trust, and collaborate on data assets across your data universe. If you are looking to implement a metadata management tool for your organization, do take Atlan for a spin. or Talk to us.
Atlan Case study: Metadata Management at WeWork
Share this article
