Atlan DemoAtlan Demo

What is a Machine Learning Data Catalog (MLDC)?

March 4th, 2021

What is a Machine Learning Data Catalog

What is a machine learning data catalog (MLDC)?

A machine learning data catalog (MLDC) is a next-generation data catalog that enables real-time data discovery and automates cataloging, crawling of metadata, and classification of PII data.
Machine learning data catalogs are an evolution from traditional data catalogs. Data Cataloging or what we at Atlan like to call Data Catalog 1.0 started with simple metadata management for IT Teams right about when data began exploding. Metadata management in this case included inventorying of data. Then came the concept of data stewardship. With it came the second generation of data catalogs that allowed a few people - data stewards in the organization to manage metadata, set & maintain governance practices, and manually catalog data.
But in an era where metadata is as big as data, machine learning data catalogs are an essential evolution that match with the capabilities of the rest of the modern data stack when it comes to being fast, flexible and scalable.
Before we get into machine learning data catalogs, let’s revise:

What is a data catalog?

Data Catalogs are a neatly organized inventory of all data assets across an organization. They seek to solve data discovery, quality & governance issues.

Read our 101 on data catalogs to get the complete lowdown on data catalogs.

"Modern machine-learning-augmented data catalogs automate various tedious tasks involved in data cataloging, including metadata discovery, ingestion, translation, enrichment and the creation of semantic relationships between metadata. These next-generation data catalogs can therefore propel enterprise metadata management projects by allowing business users to participate in understanding, enriching and using metadata to inform and further their data and analytics initiatives."

-Gartner, Augmented Data Catalogs 2019

How do machine learning data catalogs work?

    Once configured, the MLDC would:
  1. Crawl your data sources (on-premise or cloud data warehouses, lakes, databases)
  2. Understand and interpret technical metadata
  3. Create business descriptions and other such information to catalog data with context automatically
  4. Run periodic audits to verify the accuracy, quality and integrity of data
    Fundamentally data cataloging process includes most of the following steps:
  1. Discover what data exists and relations between data
  2. Enrich & annotate metadata for more context
  3. Classify & govern data
  4. Make it easier for users & other services in organization to discover, trust and use data
Machine learning data catalogs just intend to automate all of the above processes to ensure more efficiency & productivity of data teams.

Why do you need machine learning data catalogs?

Machine learning data catalogs (MLDCs) that simplify finding and inventorying siloed data assets are a crucial first step in data and analytics projects. Gartner predicts that over 60% of traditional data catalog projects that don’t use machine learning to find and inventory data will fail.

Challenges with traditional data catalogs

    Maintaining traditional data catalogs is excruciating because organizations:
  1. Generate petabytes of data every day
  2. Store data in messy, unclassified and unusable formats
  3. Handle most aspects of cataloging manually

Data consumers cannot use obsolete, unverified data to inform business decisions. With increasingly tighter regulations on data security, integrity, and privacy, that can burn a hole in the pocket. Even a minor GDPR infringement would cost either €10 million or 2% of your annual revenue, whichever amount is higher.

The consequences of using traditional data catalogs

    Legacy data catalogs require extensive manual intervention, leading to:
  1. Endless delays in projects
  2. Hefty fines for not complying with data-related regulations (GDPR, CCPA, and cohort)
  3. Difficulties in cross-team collaborations (in an increasingly distributed environment)

What are the key capabilities of a machine learning data catalog?

    According to G2, a machine-learning-powered data catalog should:
  1. Organize and consolidate data in a single repository (i.e., a single source of truth)
  2. Allow data consumers (especially business users) to search and access the data they need
  3. Let users categorize, comment and share data sets easily to improve collaboration
  4. Offer intelligent recommendations (using machine learning algorithms) to relevant data
  5. Enable user access management (UAM) for better data governance

Six essential features to expect from modern data catalogs

    While evaluating machine learning data catalogs (MLDCs), look for:
  1. Auto-cataloging
  2. Google-like semantic search
  3. Automated data lineage mapping
  4. Easy collaboration
  5. Automated quality audits and governance
  6. Intelligent recommendations

1. Auto-cataloging

A machine learning data catalog should automate tedious aspects of cataloging such as crawling metadata, classifying PII data, profiling for quality (missing values, outliers, and other anomalies). Regardless of where the data comes from (cloud warehouses, data lakes, or RDBMS), the catalog must be able to find and organize it. How does this help? Take this instance where there’s a bunch of retail sales data that are most commonly used by the revenue and marketing team to work on offers but naturally they need not have access to PII data that will automatically come with the same data assets Auto-cataloging can help identify the PII data and help mask it only to be revealed to people with authorized access.

Regardless of where the data comes from (cloud warehouses, data lakes, or RDBMS), the catalog must be able to find and organize it.

Auto-classified data sets with adequate context help data consumers
            interpret the data and use it to make strategic decisions.

Auto-classified data sets with adequate context help data consumers interpret the data and use it to make strategic decisions.

2. Google-like semantic search

Smart data catalogs like MLDCs should empower business and technical users alike to run “Google-like” searches on the metadata to address business outcomes.

Typing “Sales” on the search window should display a list of relevant data
            sets, which can be fine-tuned using filters on data type,
            source, format, and more.

Typing “Sales” on the search window should display a list of relevant data sets, which can be fine-tuned using filters on data type, source, format, and more.

The catalog should also provide one search window for all data and dashboards to improve user experience and make working with data a breeze. We have reached a day and age where every person in an organization is a data user, but it’s not possible for everyone to learn to run SQL queries. Machine learning data catalogs help make search as visual and intuitive as a search engine making data discovery easy for everyone.

3. Automated data lineage mapping

Data lineage shows the origins of data sets, how they’ve evolved through their lifecycles, and foresees the assets that will be affected by future changes. Proving lineage for building trust in data and ensuring compliance warrants tracking the transformation that data sets undergo.

Lineage also helps build better, more relevant models. So, MLDCs should be able to parse through your query logs in your data warehouses, data lakes and other data sources automatically to create a visual map of data lineage. Often lack of knowledge is a deterrent to action that can create real value. Lineage is not just about bug fixes, it’s also about giving a data user the confidence to go ahead and make changes they want in a data with clear visibility of impact. Propagation through lineage also allows to carry compliance and security tags to data flowing from particular assets.

An MLDC would track every transformation that a data set undergoes and
            represent it graphically to help users verify its lineage.

An MLDC would track every transformation that a data set undergoes and represent it graphically to help users verify its lineage.

4. Easy collaboration

In the post-pandemic era, distributed teams are here to stay. MLDCs should facilitate collaboration across teams and geographies within organizations with in-built features for in-line chats, comments and annotations, data set ratings and sharing data sets with a single URL.

Machine Learning data catalogs let data consumers discuss and collaborate within the
            platform through features like chats, comments,
            and more.

Modern data catalogs let data consumers discuss and collaborate within the platform through features like chats, comments, and more.

5. Automated quality audits and governance

Automated quality audits are an excellent way to ensure data quality, integrity and trustworthiness. Running scheduled audits to spot data regressions, data loss or distributional shifts over time can help certify data accuracy.

Running scheduled audits to verify the quality and integrity of data is a
            great way to certify its accuracy and usability.

Running scheduled audits to verify the quality and integrity of data is a great way to certify its accuracy and usability.

Tracking data usage right from the source is essential for better governance. Additionally, modern data catalogs simplify governance for IT and data stewards by providing a single dashboard to establish policies, manage access logs and requests.

 Handling access from a single window reduces delays in authorizing requests,
            which removes bottlenecks and simplifies the lives of 
            your IT teams.

Handling access from a single window reduces delays in authorizing requests, which removes bottlenecks and simplifies the lives of your IT teams.

6. Intelligent recommendations

Intelligent recommendations of other data sets that might be relevant or of interest to data consumers enhance the overall user experience.

Just like the “People also ask” and “Searches related to…” sections on Google search, this feature can show similar data sets or curate data that matches the user’s search criteria to increase the value derived from data.

Quick recap

  1. Machine learning data catalogs (MLDCs) enable real-time data discovery and automatic cataloging of data sets with adequate context.
  2. With MLDCs, organizations can build a single source of truth for all data, track lineage, search and access the right data via a single dashboard.
  3. Modern, augmented data catalogs facilitate improved collaboration within the organization, empower all data users to make data-driven decisions, simplify governance, and facilitate data democratization.
  4. While evaluating data catalogs, look for automated ingesting, inventorying, tagging, profiling, lineage mapping, and enrichment of data sets.

Can machine learning data catalogs increase business value?

Machine learning data catalogs can definitely increase business value. They effectively contribute towards reducing speed to insights, greater business engagement, retention of great data talent - and of course, better utilization of data and the modern data stack.

Are you looking to deploy a machine learning data catalog that will be the perfect fit for your business requirements? Your business requirements can range from data democratization initiative to data governance strategy setups, to something as simple as increasing efficiency of your data team - use this step-by-step checklist to create your customized evaluation criteria framework. This way you get a clear picture - if the machine learning data catalog under consideration is matching your immediate and long term business needs.

    Enterprises using Atlan’s machine-learning-augmented data catalog have reported:
  1. Up to 60X speed to insight
  2. 70% greater business engagement
  3. ROI realization within two weeks
Ebook cover - data catalog primer

Data Catalog Primer - Everything You Need to Know About Modern Data Catalogs.

Adopting a data catalog is the first step towards data discovery. In this guide, we explore the evolution of the data management ecosystem, the challenges created by traditional data catalog solutions, and what an ideal, modern-day data catalog should look like. Download now!