Unity Catalog: An overview
Permalink to “Unity Catalog: An overview”The catalog is where an AI agent finds the right table to answer a business question. Without it, the agent guesses. Atlan holds that context outside the Databricks runtime, which is what the five limits below come down to. Unity Catalog handles the inside-Databricks half well, but an agent’s questions rarely stop at one platform’s boundary. When working with Databricks, Unity Catalog lets you manage access from one place, explore lineage, and search and discover data in your account, yet end-to-end data governance, lineage, and metadata enrichment all stop at the Databricks boundary.
There are two Unity Catalogs, and they share a name without sharing a governance surface. The managed service inside Databricks is the one this page is about. The open source Unity Catalog project at unitycatalog.io implements the catalog API across engines and table formats, and describes its governance as strong authentication, secure credential vending, and asset-level access control. Asset-level is the operative phrase: the open source project answers interoperability, and the row-level and column-level controls below belong to the managed service.
Understanding the five primary limitations of the Unity Catalog
Permalink to “Understanding the five primary limitations of the Unity Catalog”Each of the five limitations below lands on a different workflow, and each one exists for a reason worth knowing before you plan around it.
1. End-to-end governance is limited to the Databricks boundary
Permalink to “1. End-to-end governance is limited to the Databricks boundary”Unity Catalog is designed to have authority within a Databricks account and the Databricks runtime. It doesn’t control the data itself. When data flows to other connected systems, most of the governance-related features don’t work. That covers visualization tools, external query or compute engines, and orchestrators.
There are exceptions to this where you can use some of the deep Unity Catalog integrations to synchronize governance metadata with external tools, but these are few and far between. There are also Unity Catalog’s Open APIs, but for them too, the enforcement still rests with the Databricks runtime.
That external access is real, and it has widened. According to Databricks’ external access documentation, the Unity Catalog Iceberg REST Catalog endpoint carries read, write, and create access for Apache Iceberg clients to managed Apache Iceberg tables, with credential vending controlling which cloud storage locations those clients reach. OpenSharing, which Databricks introduced in June 2026 as the next evolution of Delta Sharing, covers reads outside the account. The bytes travel, and since 2026 they travel in both directions.
The fine-grained policy does not travel with them. According to Databricks’ row filter and column mask reference, you cannot use the Iceberg REST catalog or Unity REST APIs to access tables with row filters or column masks, and OpenSharing providers cannot share tables carrying table-level row filters or column masks. One path does get through, and it makes the point rather than closing it. Databricks’ cross-engine attribute-based access control, in Beta, lets an external Iceberg client read a filtered table: Unity Catalog evaluates the policy during scan planning and returns a scan plan already narrowed to the rows that principal may see. Reads only. Specific client versions, external data access enabled on the metastore, and the EXTERNAL USE SCHEMA privilege granted. The policy never leaves Databricks. An outside engine that wants filtered data has to ask Databricks to do the filtering.
The perimeter has an internal seam too. Databricks’ metastore documentation states that you must have one metastore for each region in which your organization operates, and that users must be on a workspace attached to a metastore in their region. A multi-region estate therefore runs more than one metastore and gets more than one graph, with access across them handled by sharing rather than by one catalog.
For organizations running hybrid or multi-platform data stacks, this boundary creates a persistent governance blind spot.
2. External data lineage capabilities and limitations
Permalink to “2. External data lineage capabilities and limitations”The same boundary restriction applies to lineage metadata; however, solving the lineage limitation is slightly easier because it has more integration paths to bridge the Databricks boundary. There are many tools, such as dbt, that have deep integrations with Unity Catalog, while others like Airflow and Fivetran can push lineage into Unity Catalog via the API.
That said, there are clear limitations in what you can do with the lineage you push to Databricks. The depth and automation don’t compare with native lineage, and some additional features, such as editing lineage metadata and syncing metadata with external systems, aren’t supported.
For tools like Power BI, Tableau, and Salesforce, you can register them as lineage sources via the External Lineage API or Catalog Explorer, but this requires manual handling. Even with the deeper integrations, granular end-to-end lineage across your data stack remains a challenge.
3. Metadata enrichment has improved over time, but is still mostly limited to Databricks
Permalink to “3. Metadata enrichment has improved over time, but is still mostly limited to Databricks”Features like governed tags, Unity Catalog metrics, automated data classification, and AI-driven documentation help with governance and metadata enrichment for data catalogs. That said, many of these features are still in the early stages of release. While they’re great additions, these features are limited to the Databricks ecosystem, all within Unity Catalog. The same limitations apply to discovery and lineage, too.
Some of these limitations, especially related to querying data through other systems, are mitigated by Lakehouse Federation, which has its own quirks and limitations.
4. Delta-centric design creates friction for diverse data ecosystems
Permalink to “4. Delta-centric design creates friction for diverse data ecosystems”Early on, Databricks created a table format called Delta Lake for the object storage layer. Databricks’ subsequent features were designed specifically around this format, but outside of a few tools, Delta Lake wasn’t widely adopted. Newer table formats, such as Iceberg, got much wider adoption.
As many key Databricks features rely on Delta, this dependency doesn’t translate well when working with external systems. The governance features don’t work quite the same way; partition management is also quite different.
This is the limitation that has moved most. According to Databricks’ May 2026 announcement, Unity Catalog managed Apache Iceberg tables, foreign Apache Iceberg tables, and Apache Iceberg v3 all reached general availability, and the Iceberg REST Catalog lets engines outside Databricks read and write those managed tables directly. Treat the format gap as largely closed, and do not buy a third-party product to solve it. What has not moved is where enforcement sits. Put a row filter or a column mask on one of those tables and every external read has to route back through Databricks to be filtered first. The table travels. The rule that protects it does not.
5. Key security features need specific Databricks runtimes
Permalink to “5. Key security features need specific Databricks runtimes”The availability of security features depends on many factors, such as the type of Databricks Object or the Databricks Workspace you’re working with. Some of these issues relate to high-level governance, some to identity, and some others to access control.
Unity Catalog features like Row-level Security (RLS) and column masking policies combined with Attribute-based Access Control (ABAC) need DBR 16.4 or higher if you’re using dedicated compute, or serverless compute.
How can you solve Unity Catalog’s limitations?
Permalink to “How can you solve Unity Catalog’s limitations?”Some of the Unity Catalog limitations mentioned in the previous section, such as metadata enrichment and runtime requirements, are already handled by Databricks, so they will be resolved over time.
Others are structural and will not be fixed in a future release. They come down to the scope and the surface area of governance, lineage, and discovery.
Unity Catalog limitations in severity order, and what actually closes each one
Ordered by the split above: the structural limits first, because those are the ones Unity Catalog releases will not close, then the ones Databricks is already closing. The last three rows are the honest test of this table, because their answer is not a purchase.
| Limitation | Structural or maturity | Why Unity Catalog cannot close it alone | What closes it |
|---|---|---|---|
| Policy enforcement stops at the Databricks perimeter (limitation 1) | Structural | Enforcement is bound to the Databricks runtime, not to the data. A table carrying a row filter or column mask cannot be read through the Iceberg REST catalog, cannot be shared at table level through OpenSharing, and reaches an outside engine only when Databricks filters it server-side first. | A neutral context layer that carries policy context, ownership, and classification into every tool that reads the data, and syncs both ways with Unity Catalog. |
| Lineage is scoped to one metastore, and Databricks requires one metastore per region (limitation 2) | Structural | A multi-region estate runs more than one metastore and gets more than one lineage graph. External assets need manual registration through the External Lineage API and do not match native depth. | Column-level lineage stitched across every source, editable, and pushed back into Unity Catalog. |
| Business context does not propagate: domains, tags, certifications, glossary | Structural | Constructs authored in Unity Catalog have no equivalent in the tools downstream, so the business language breaks at the boundary. | One glossary and one domain model, with two-way propagation of tags and classifications across the estate. |
| Single-stack lock-in on the context itself | Structural | Every policy, tag, certification, lineage edge, and glossary term authored in Unity Catalog is expressed in one vendor’s model and enforced by one vendor’s runtime. Adding an engine means re-authoring it. | Holding context in a neutral context layer on open formats, so the definitions outlive any single platform decision. |
| Estate coverage beyond Databricks: warehouses, on-premises systems, BI, streaming, SaaS | Structural | Lakehouse Federation gives governed, read-only access to foreign catalogs. OpenSharing (the superset that succeeds Delta Sharing) recipients are read-only, and sharing across clouds or regions can incur cloud egress fees. The open source Unity Catalog project covers a narrower surface than the managed service. | Native connectors to the whole estate, so coverage stops depending on what one platform federates. |
| Delta Lake first, Iceberg second (limitation 4) | Maturity, largely closed | Databricks features were built around Delta. Managed Iceberg tables, foreign Iceberg tables, and Iceberg v3 reached general availability in May 2026, with credential vending on the same track. | Databricks has already closed most of this. Track the release notes; do not buy a workaround for it. |
| Metadata enrichment confined to Unity Catalog (limitation 3) | Maturity, improving | Governed tags, Unity Catalog metrics, automated classification, and AI-generated documentation are real features, and they apply to Databricks assets only. | Databricks is closing the Databricks half. Only the estate-wide half needs a layer above it. |
| Security features gated on Databricks Runtime 16.4 or higher (limitation 5) | Self-resolving | ABAC with row-level security and column masking needs DBR 16.4 or higher on dedicated compute, or serverless compute. | Upgrade the runtime, or move to serverless compute. No third-party product is involved. |
- Fragmented governance with a strict Databricks boundary results partly because the enforcement is tightly coupled with the compute layer of Databricks, and not the data itself. This means that when data flows to other tools, such as BI tools and orchestration engines, Unity Catalog-specific enforcements for access control, data masking, etc., don’t translate well.
- Siloed and fragmented lineage across regions and other tools in the data stack, as lineage is scoped at the metastore level and doesn’t reliably synchronize with other platforms. A few integrations carry lineage deeply. Most require manual registration and don’t match the depth of Databricks’ native lineage.
- Lack of context propagation and synchronization between Unity Catalog and other tools leads to a broken business language and mixed signals for data quality, trust, and security when constructs like domains, tags, and certifications don’t translate, and when the semantic definitions an agent reads live in a different model from the tables they describe.
The cost of single-stack lock-in
Permalink to “The cost of single-stack lock-in”Single-stack lock-in with Unity Catalog is a context problem, not a storage problem. Delta Lake is open, managed Iceberg tables reached general availability in May 2026, and external engines now read and write them through the Iceberg REST Catalog. The bytes are portable. The policies and the meaning are not. Every access rule, every governed tag, every certification, every lineage edge, and every glossary term you author inside Unity Catalog is expressed in one vendor’s model, enforced by one vendor’s runtime, and scoped to a metastore in one region.
That cost arrives the day the estate stops being Databricks-only: a warehouse the finance team already runs, an on-premises system nobody is migrating, a BI tool with its own permission model. Each addition means re-authoring context that already exists somewhere else. A neutral context layer inverts the dependency. The context lives outside the runtime that happens to be reading the data, so adding an engine becomes a connector rather than a re-implementation.
All of these issues stem from the same absence: a neutral context layer that consolidates context from every source in your organization and stitches lineage across every system that organization works with.
Atlan is that layer. It brings structural, policy, and lineage context into a single place. Let’s look at how Atlan helps mitigate Unity Catalog’s limitations.
How does Atlan’s enterprise context layer extend beyond Databricks?
Permalink to “How does Atlan’s enterprise context layer extend beyond Databricks?”Inside Databricks, Unity Catalog already covers discovery, lineage and who may see what. Outside it, every tool keeps its own version of the definitions, and nothing holds the estate to one set.
Atlan closes that gap as a neutral context layer: built on open formats and governed estate-wide, owned by the organization, and independent of any single platform’s runtime.
Atlan uses a standardized metadata schema, and a broad range of connectors to ingest metadata from wherever you need it. Rather than just using Unity Catalog, you bring all data, including that from Unity Catalog, and manage it all in Atlan’s Enterprise Data Graph.
Built in close partnership with Databricks, Atlan layers on top of Unity Catalog to extend its value across your broader data ecosystem.
Key capabilities include:
- A mature ecosystem of connectors supporting a wide variety of data tools.
- Domain-based organization, persona-based search and discovery.
- End-to-end granular data lineage with the ability to edit and synchronize it with other systems.
- Two-way synchronization and propagation of attributes like tags, policies, and classifications.
- Automated data quality monitoring leading to better and live trust signals.
- A business glossary that works across all the tools in the data stack of your organization.
- Policy context that applies to AI assets as well as data assets, so agents and models inherit the same rules as the tables beneath them.
The Enterprise Data Graph, and the Context Lakehouse beneath it, are what make that context usable: lineage impact analysis, tag and classification propagation, live quality signals, and estate-wide search. It covers lineage, data quality, policy context, security and observability, precisely the areas where Unity Catalog’s design creates friction for organizations using multiple data systems and platforms.
Real stories from real customers using Atlan with Databricks
Permalink to “Real stories from real customers using Atlan with Databricks”Activating Databricks metadata with Atlan’s unified context layer
Permalink to “Activating Databricks metadata with Atlan’s unified context layer”
Activating Databricks metadata with Atlan's unified context layer
"More than Databricks, we needed a platform for innovation to stay ahead of our competitors. We might know what we need right now, but if the market is moving in a new direction, with AI and ChatGPT, for example, we need to have an answer for that, and the opportunity to try these tools in our data catalog. That's what I really liked about Atlan."
Jorge Plasencia, Data Catalog & Data Observability Platform Lead
Yape
Listen to the podcast: Why Yape chose Atlan to govern Databricks
53% less engineering workload and 20% higher data-user satisfaction
Permalink to “53% less engineering workload and 20% higher data-user satisfaction”
53% less engineering workload and 20% higher data-user satisfaction
"Kiwi.com has transformed its data governance by consolidating thousands of data assets into 58 discoverable data products using Atlan. 'Atlan reduced our central engineering workload by 53% and improved data user satisfaction by 20%,' Kiwi.com shared. Atlan puts ownership, contracts and open data-quality issues in one place, which is what makes governance workable day to day across teams."
Data Team
Kiwi.com
Listen to the podcast: How Kiwi.com Unified Its Stack with Atlan
Moving forward with a neutral context layer for your data and AI ecosystem
Permalink to “Moving forward with a neutral context layer for your data and AI ecosystem”Unity Catalog is a solid built-in governance, lineage, and cataloging tool for Databricks-heavy environments. What it lacks is multi-platform support, and that produces the siloing and fragmentation this kind of tool exists to prevent. Some of the other limitations are due to product maturity and will be mitigated by future releases of Unity Catalog.
A context layer that aggregates and activates context across your data stack, and that no single platform owns, is what closes the remaining gap. Such a layer is the foundation for higher-order capabilities, such as lineage impact analysis, tag and classification propagation, and organization-wide search and discovery. With its current limitations, Unity Catalog can’t address these challenges on its own.
Atlan’s neutral context layer fills that gap: context from Unity Catalog and every other tool in the stack, in one enterprise-governed layer that activates policy context, lineage, and discovery at scale.
See how a neutral context layer extends your Databricks stack.
Book a personalized demoFAQs about Unity Catalog limitations
Permalink to “FAQs about Unity Catalog limitations”1. What are the key limitations of Unity Catalog?
Permalink to “1. What are the key limitations of Unity Catalog?”The key limitations are related to support for and integration with other tools in the data stack. For governance, lineage, and discovery, the biggest limitation is clearly the inability to fully synchronize this metadata with other tools.
2. Does Unity Catalog work with external tools?
Permalink to “2. Does Unity Catalog work with external tools?”Yes, but in a limited capacity. Databricks’ Open APIs and Lakehouse Federation help it connect to various tools, but the governance, lineage, and discovery functionality is quite limited and still bound by the Databricks perimeter.
3. Does Unity Catalog support data lineage across platforms?
Permalink to “3. Does Unity Catalog support data lineage across platforms?”Yes, but to a limited extent. You can set up external lineage ingestion manually using Unity Catalog or the External Lineage API, but coverage is quite limited and typically doesn’t go down to the granular level.
4. Can Unity Catalog govern data outside of Databricks?
Permalink to “4. Can Unity Catalog govern data outside of Databricks?”While some synchronization capabilities exist, Unity Catalog currently cannot govern or enforce policies outside Databricks. All policies are enforced at the Databricks compute layer. Lakehouse Federation gives governed, read-only access to foreign catalogs, and external locations cover cloud object storage. Neither extends enforcement to on-premises systems or to non-Databricks engines.
5. What Databricks Runtime version is needed for ABAC?
Permalink to “5. What Databricks Runtime version is needed for ABAC?”Attribute-based Access Control (ABAC) with granular row-level security (RLS) and column masking requires Databricks Runtime 16.4 or higher on dedicated compute. You can use serverless compute, and this limitation goes away. Also, remember that older runtimes can’t access tables secured by ABAC policies.
6. Does open source Unity Catalog have the same limitations?
Permalink to “6. Does open source Unity Catalog have the same limitations?”The open source Unity Catalog project at unitycatalog.io implements the catalog API across engines and table formats including Delta Lake and Apache Iceberg. It describes its governance as strong authentication, secure credential vending, and asset-level access control. That is a different surface from the managed service: it answers interoperability at the asset level, not row-level and column-level enforcement or estate-wide policy context.
7. Can external engines read Unity Catalog tables through the Iceberg REST Catalog?
Permalink to “7. Can external engines read Unity Catalog tables through the Iceberg REST Catalog?”Yes. Databricks’ Open APIs include an Iceberg REST Catalog endpoint with read, write, and create access for Apache Iceberg clients to managed Apache Iceberg tables, and credential vending controls which cloud storage locations those clients reach. OpenSharing, which extends Delta Sharing rather than replacing it, covers cross-organization reads. One limit matters most: the documentation states you cannot use the Iceberg REST catalog or Unity REST APIs to access tables that carry row filters or column masks. A Beta cross-engine path lifts that for reads, and it works by having Unity Catalog evaluate the policy and return a pre-filtered scan plan, not by handing the policy to the engine.
8. How do you avoid single-stack lock-in with Databricks?
Permalink to “8. How do you avoid single-stack lock-in with Databricks?”Keep the context outside the runtime. Author policies, lineage, domains, and glossary terms in a layer that syncs both ways with Unity Catalog, keep tables in open formats such as Delta Lake and Apache Iceberg, and use OpenSharing or the Iceberg REST Catalog for external reads. Then the tables stay portable, and so do the definitions that make them usable.
