Databricks Unity Catalog + Atlan: A Single Metadata Control Plane for Your Enterprise

Updated July 08th, 2024

Share this article

Databricks Unity Catalog is a technical catalog that manages data architecture, integration, and performance.

Bringing Databricks Unity Catalog together with Atlan vastly expands the potential for accommodating diverse data and AI use cases. You get an interconnected data estate that supports diverse humans of data — architects, engineers, data scientists, business and financial analysts.

This article explores how they work together.

Table of contents #

  1. How does Databricks Unity Catalog differ from Atlan?
  2. Atlan: A unified layer for all metadata, not just technical metadata
  3. Integrating Unity Catalog and Atlan: 3 factors to consider
  4. Databricks Unity Catalog + Atlan = Better together
  5. Related reads

How does Databricks Unity Catalog differ from Atlan? #

Databricks Unity Catalog is a technical catalog, whereas Atlan is a metadata control plane, i.e., a comprehensive, unified layer that connects with every tool in your data stack.

To understand how the two platforms are different and explore their synergies, let’s look at the anatomy of technical catalogs and the impact of emerging technologies on the data stack.

Technical data catalogs: Catering to technical personas and use cases #


Technical data catalogs aim to solve specific problems in the data ecosystem. The most popular ones include Glue (by AWS), Purview (by Microsoft), Unity Catalog (by Databricks), and Polaris (by Snowflake).

Technical catalogs like Unity Catalog help expose metadata about data assets, products, lineage, tags, policies, and more.

However, leveraging the metadata from technical catalogs is still complex. They’re designed for data practitioners with technical skills — developers, engineers, and architects. For instance, data stewards using Unity Catalog must have technical expertise to write scripts and use the CLI to configure governance workflows (policies, access control, etc.).

Meanwhile, the rest of your team — analysts, marketers, decision-makers — need more than just a technical catalog. They need context, collaboration, and governance, which in turn, requires a unified metadata control plane.

A unified control plane for data and AI that goes beyond technical catalogs #


Modern data environments are diverse and multi-cloud, catering to diverse teams and connecting numerous tools within an organization.

More so with the advent of AI, the use cases that data teams want and are expected to drive are ballooning. Meanwhile, metadata is growing in significance, as high-quality metadata is central to powering AI and LLM applications.

In this landscape, a data catalog must act as a unified control plane that’s built for diversity:

  • Diversity in data assets: Data assets go beyond rows and columns to include databases, schemas, dashboards, requests, pipelines, READMEs, notebooks, code snippets, metrics, and more. Each asset has its own metadata (technical, human, producer), and together they create a 360-degree profile for that asset.
  • Diversity in humans of data: Data teams have diverse personas — analysts, engineers, data scientists, business analysts, and financial analysts. Each persona has their own vocabulary, tools, and use cases with different curation and customization needs. This diversity means that we need a layer capable of delivering metadata in a personalized manner.
  • Diversity in data tools: With diversity in data roles, you get diverse tools and platforms — metadata lakes, cloud data warehouses, ETL tools, BI platforms, data processing frameworks, orchestration engines, communication tools, and more. Data practitioners want context where they are, when they need it — in a flow.
  • Diversity in use cases: Diverse assets, personas, and data stack lead to diverse data, analytics, and AI use cases. Use cases also vary depending on the industry and your organization’s data maturity.

A unified control plane would act as a single pane of glass to consume and curate all kinds of metadata. As a result, every individual, regardless of their role, can efficiently access and utilize the metadata they need.

The control plane would also serve as the “nervous system” of a diverse data stack infrastructure, integrating and managing these diverse tools seamlessly. It would embed within your workflows to reduce switching costs, eliminate distractions, and improve collaboration.

Atlan: A unified layer for all metadata, not just technical metadata #

Atlan provides a consistent layer that helps access and manage metadata across your enterprise, regardless of the underlying platforms.

Atlan supports use cases across teams, personas, and organizations, offering the following benefits:

  • Trusted data for confident decisions
  • Assured data security and regulatory compliance
  • Accelerated data and AI innovation
  • ROI on your tech stack

Let’s look at some examples of how Atlan delivers each of its benefits.

Trusted data for confident decisions #


Unity Catalog works with your existing data catalogs, data storage systems and governance solutions to centralize data access and improve collaboration.

When you integrate Databricks Unity Catalog and Atlan, you get:

  • A business glossary that brings context where you work. For instance, a Chrome extension in BI dashboards.
  • Data product marketplace with a complete repository of trusted data products, each owned and maintained by their domains.
  • Proactive impact analysis and issue alerting powered by cross-system, column-level actionable data lineage for a single pane of glass.

Assured data security and regulatory compliance #


Databricks Unity Catalog is equipped with Databricks’s enterprise-grade security. So, you can configure fine-grained control on rows and columns using a low-code interface.

Atlan enhances this with:

  • Auto-propagation of data classification and tags for Databricks assets (along with other essential metadata, such as data asset name and description).
  • A centralized, no-code Policy Center that connects Atlan Policies to Databricks assets, so that you can stay on top of all things governance for your entire data estate.
  • A no-code setup for data stewards to configure access policies.
  • Data contracts that embed data governance guardrails into the data producer tools and workflows.
  • Governance by exception — alerting data stewards whenever a data asset doesn’t comply with your data governance policy.

Accelerated data and AI innovation #


Combining Databricks Unity Catalog and Atlan can improve data search, discoverability, and context for your data estate with:

  • An intuitive UX that powers natural language search across your data universe — Tableau dashboards, dbt models, Airflow DAGs, etc.
  • Embedded collaboration that brings context where you work. For instance, Atlan’s Chrome extension brings Databricks context directly into BI tools used by business users.
  • Intelligent automation that scales data asset documentation, tag propagation, PII data classification, asset recommendations, and more

ROI on your tech stack #


Atlan’s open API helps bring in metadata from any tool in your data stack — for e.g. runtime metrics from data processing engines, or usage metrics from BI tools.

Atlan’s real-time automated metadata capture layer is configurable, extensible, and provides an overall view of data estate and usage.

This helps in optimizing the ROI of your tech stack by letting you:

  • Track the popularity of each data asset with usage metadata to spot and review underutilized assets.
  • Analyze popularity metrics to allocate resources better and optimize your investment in tools and technologies.
  • Discover related data assets and decrease the time-to-insight, while driving decision-making.

Integrating Unity Catalog and Atlan: 3 factors to consider #

As you integrate a technical catalog like Databricks Unity Catalog with Atlan, you must consider three factors for a successful setup:

  • Tech stack complexity: The number of tools in your data stack are exploding. How do you manage the complexity and scale your infrastructure while ensuring its useful to everyone?
  • Cross-functional complexity: The personas and business teams involved in data are growing. How do you cater to such diverse curation and customization needs?
  • Use case complexity: Data team use cases vary depending on your data stack, organizational maturity, industry, and ability to adopt emerging technologies (like GenAI). How can you support each use case and make sure your data estate can accommodate more at scale?

Databricks Unity Catalog + Atlan = Better together #

By combining the technical prowess of Databricks Unity Catalog with Atlan’s metadata control plane, you can create a powerful data ecosystem that fosters trust, empowers decision-making, and drives innovation across your organization.

For instance, with Databricks and Atlan, General Motors now has 50+ data and AI use cases in production, 2,100+ users accessing insights monthly. Additionally they’ve seen an acceleration of time-to-insight from 28 days to under 3 hours, and a substantial $330 million boost to their bottom line

using Atlan for end-to-end visibility from the cloud all the way back to our on-prem

Using Atlan for end-to-end visibility from the cloud all the way back to our on-prem - Source: Brian Ames, Head of AI Center at GM using Atlan and Databricks for data and analytics.

Eliminate fragmented data and siloed insights and drive data and AI use cases with Databricks Unity Catalog and Atlan.

Share this article

[Website env: production]