Is the Data Catalog Finally Dead?

Data catalogs have always had three fundamental problems: creation, consumption, and learning. Every one of them gets harder in the AI era, not easier.

Creation

Someone had to populate the catalog — descriptions, business terms, domains, owners, all kept current. At enterprise scale, that was never possible; the data estate always changed faster than humans could document it.

Now the gap widens: AI needs far more context than it ever did for people, and humans cannot supply it. Every asset, relationship, decision, and policy a growing population of agents will touch has to be documented — by a workforce that could never keep up with the old demand, let alone this one.

Consumption

Even when the context existed, people had to remember to go looking for it: leave the workflow, open the catalog, translate a question into catalog language. We spent years reducing that friction — automation, lineage, Slack integrations, natural-language search — and none of it changed the model.

Now it breaks in a new way: an agent reasoning inside Claude or Cursor has no interface into the catalog at all. A search box and a set of pages were built for human eyes. An agent needs to reach the same context as tools it can call, mid-reasoning, or it never sees it.

Learning

Success was measured by coverage — how many assets have descriptions, owners, lineage — and coverage cannot tell you whether the context actually worked. When a definition was wrong, missing, or right-but-misleading, nothing traced the bad answer back to the gap that caused it. Humans mostly stayed quiet, so the catalog never heard about its failures.

Agents will not. They will hit the catalog thousands of times a day, and we will need a live sense of what they can see, what they can’t, and what is missing — the exact thing coverage was never able to tell us.

The agentic catalog is what happens when agents are creating context, consuming accurate context quickly and efficiently, and improving context with a learning loop.

The first change is on the production side. Humans cannot produce the volume of metadata AI requires, so the work moves to Context Agents. Instead of starting from an empty field, an agent reads the evidence already sitting in the estate — lineage, SQL and query history, usage patterns, column metadata, dbt logic — and writes a grounded first draft. Humans set the standards, inspect the evidence, resolve ambiguity, and certify what can be trusted.

This is why “AI-powered catalog” is too small a frame — it sounds like an old catalog with a chatbot bolted on. An agentic data catalog changes the operating model:

• Agents create and maintain first-pass metadata from evidence.
• Humans move from writing every field to governing the system.
• The catalog becomes a supply of AI-ready context that keeps pace with the estate.

The oldest bottleneck in cataloging can finally be attacked at the scale of the estate — and the quality of that context compounds: each layer of signal makes the next one better.

Can agents really document a data estate?

We have been asking this for more than two years. We launched Atlan AI in 2023, built a useful AI-powered catalog, and watched quality plateau at roughly 70%. The lesson: this takes more than a better prompt. To be accurate, AI needs rich signals — lineage, query history, usage, relationships — and Atlan stored all of it in systems built to serve a human interface, not to be queried at the speed and scale an agent needs.

So in 2025 we rebuilt the foundation as a context lakehouse: the catalog’s knowledge landed as open Apache Iceberg tables in the customer’s own cloud, queryable by anything that speaks SQL. Then we built the agent harness, and rebuilt it, several times. Early this year — stronger models, deeper harnesses, a far more rigorous context-quality pipeline — it started working.

Our first Context Pioneers cohort is the early evidence. Across 50 enterprises, Context Agents generated 1.03 million descriptions and 15,000 READMEs in three weeks, saving more than 110,000 hours; 87% of participants said the output matched or beat what a person would have written. These are Atlan-reported program results, but the individual customer accounts reveal what changed underneath the numbers. See the cohort results and participant list.

Agents recover knowledge nobody had written down

Query-pattern analysis turned real usage into business-friendly questions, common joins, and important filters. What surprised teams was not the volume — it was the quality, and the knowledge that surfaced from signals no one had ever written down.

Keith Guyett

Atlan Context Pioneers

"We were stunned with the quality of the content that was produced by these AI agents."

Izabela Wilczyńska

Data Governance Manager, PayU

"The kind of knowledge that usually lives in one person's head or a Slack thread suddenly became visible across thousands of assets."

Swatilekha Saha

Data Architect, DAT Freight & Analytics

"That's when I realized this was knowledge synthesis, not documentation."

Trust came from traceability

The old objection to automated metadata is obvious — what happens when the machine invents a plausible but wrong definition? What changed skeptics’ minds was not scale but provenance: every claim traced back to evidence they could inspect.

Cody Brees

Data Governance Lead, The Bancorp

"The context traced back to something I could point at rather than arriving as an assertion."

Simon King reported the same across 4,899 assets: because every description was grounded in lineage and real query behavior, his team could show why a field was defined the way it was. Vineet Shukla’s team saved 8,412 hours with domain-specific context that needed only minor edits, and Amulya Sagi’s lean governance team finally cleared its enrichment bottleneck. Read Simon’s post.

The data steward transforms into a context engineer

The natural fear is that if agents can perform the work of cataloging, the people who have spent years doing that work become less important.

The opposite is more likely.

When agents take on first-pass enrichment, human expertise moves up the stack. The job is no longer to fill thousands of empty fields. It is to decide what good context looks like, determine which sources should be trusted, resolve conflicting definitions, test agent output, encode policy, and identify the exceptions where business judgment matters.

Leonardo Vinicius Ruivo found that automation gave governance professionals more time to validate information, improve context, and make decisions requiring genuine business knowledge.

Leonardo Vinicius Ruivo

Data Engineer, Porto

"It allows Data Governance professionals to spend more time validating information, improving context, and making decisions that require real knowledge of the business."

David Nishimura offers an important qualification: the output was not perfect. But that imperfection did not invalidate the model. It clarified the human role. Documentation stopped looking like a periodic cleanup project and started looking like a continuously enriched product.

David Nishimura

Atlan Context Pioneers

"The output wasn't perfect, but it demonstrated how governance, metadata, and AI are converging faster than many organizations realize."

This is the next phase of purpose for the data governance community.

The data steward becomes a context engineer: part curator, part evaluator, part system designer. Their responsibility is not a completed catalog. It is the continuing quality of the AI-ready data foundation on which humans and agents rely.

People who have spent years working on metadata are unusually well prepared for this role. They understand that enterprise meaning is messy. They know a definition can be technically accurate and still wrong for a particular audience. They know provenance, ownership, policy, and trust are not decoration around the data. They are what make the data usable.

When I renamed Metadata Weekly to Context & Chaos, I wrote that our original metadata bet had been too small. Metadata had not merely become important to the data stack. It was becoming the foundation of enterprise AI.

The people who built the metadata layer were context engineers before the industry had a name for the discipline.

Live webinar · September 10, 11 AM ET

Traditional data cataloging is dead. See what comes next, live.

Can AI really document your data estate? What does an agentic data catalog look like, and how does governance evolve in an AI-first world? An agentic data catalog has generated 2M+ descriptions and saved 210,000+ hours across 200+ organizations. Join the teams who ran it in production.

Prukalpa Sankar

Co-founder, Atlan

Zhenni Hu

Data Governance, Mastercard

Keith Guyett

Solutions Architecture, Sophos

Gary Deffendall

VP Data Governance, SouthState Bank

Nandini Tyagi

Strategic Initiatives, Atlan

Rupal Sumaria

Head of Data Management, ASOS

Teams from

Register now →

The second change is on the consumption side. The catalog’s primary consumer may no longer be a person who searches and reads. It is now AI.

The interface is now AI

The way people reach the catalog has changed. They no longer type keywords into a search box — they ask a question in plain language, wherever they already work: inside Claude or an IDE through MCP, or inside Atlan’s own conversational AI. And increasingly the one asking isn’t a person at all, but an agent reasoning on their behalf. The interface, for humans and agents alike, is now AI.

The context lakehouse: the foundation that makes AI work

None of this runs on a catalog built for human eyes. What makes it work is the context lakehouse — the foundation that turns the catalog into AI-ready data and exposes it through every interface AI needs.

Every asset is stored with its own vector embedding, so retrieval works by meaning, not keyword matching. That is what powers conversational AI: you ask in plain language, and a reasoning agent plans the retrieval, searches by meaning, and returns one grounded answer instead of a list of links.

And the same store answers three ways over one foundation — vector search by meaning, graph traversal across lineage, and SQL for exact rows — blending semantic and keyword search when a question needs both.

The context lakehouse is why any AI, on any interface, can do real work on top of Atlan.

MCP: the interface for agents

For an agent, that context arrives through MCP. Atlan MCP exposes the catalog’s capabilities — search, lineage, glossary, data quality, governance — as tools an agent can call directly, under the same permissions that already govern the catalog.

But MCP is only the interface, not the intelligence. A reliable interface, for an agent, means one thing: accurate, relevant, fast, and token-efficient — tokens are the currency, and a retrieval layer that wastes them degrades every step that follows. The hard problem is context retrieval, in two dimensions: breadth and depth.

The breadth problem: where should the agent look?

A catalog holds millions of assets, and the same word can be a metric, a glossary term, a dashboard, or an entity. Search too broadly and the agent retrieves plausible but wrong context; with overlapping tools it wanders, retries, and lands in the wrong domain.

Solving breadth is not more tools — it is a bounding box around the question: which domain and source are in scope, is this a lookup or an investigation, which assets are certified or owned by the right team, and when a term is ambiguous, whether to retrieve more or ask.

Most of the industry answered breadth by wrapping existing APIs in an MCP server. Agents do not think in API endpoints; they think in tasks. So we pulled 65,763 real agent queries from 200+ organizations and bucketed them by the job each was trying to do, then built the tools around the jobs. And the jobs turn out to be more human than “metadata” suggests.

People search by association — the table someone built, at some point, for some purpose — more than by name. That is what semantic search is for.

Some questions can’t be broken into a search at all. They are open-ended investigations — the kind you would hand to an analyst, not a search box. For those, the agent hands the whole question to Atlan’s own reasoning harness, which plans the work, queries the context layer, iterates, and returns one grounded answer — so the client’s model never explores the catalog call by call.

The depth problem: how much should the agent retrieve?

Once the agent finds the right neighborhood, it faces the opposite risk. Retrieve too little and the answer omits a column, a lineage edge, or a policy; retrieve too much and the response floods the context window, raises cost and latency, and buries what matters. Early MCP responses ballooned into hundreds of thousands of tokens because a tool returned every field — the model would truncate, lose context, retry, and make it worse. The fix is progressive disclosure: start with a small, ranked, high-confidence result, make the true scope visible, and let the agent drill into exactly what it needs.

Some of the biggest wins were unglamorous. Our search tool used to carry about 2,500 tokens of overhead before it returned anything useful; rewriting the tool documentation cut that to 400, and rebuilding the retrieval engine underneath dropped median search latency from roughly half a minute to under seven seconds.

Lineage was the sharpest case. We watched a single full downstream-lineage trace burn roughly 10% of a month’s credit allocation and come back with no answer — the graph was simply too large to serialize into a context window. So the lineage tool stopped returning the whole graph. Now it infers what the question is actually about — the start asset, the direction, the type of thing at risk — and returns only the relevant subgraph.

Most recently we saw more than 90,000 tool calls in a week, from 540 users across 135 organizations, through Claude Code, Claude, Cursor, and customers’ own agents. Every tool runs against an evaluation suite that asks three questions — did the agent pick the right tool, call it with the right arguments, complete the task — and on the current baseline it selects the right tool 88.9% of the time and fills in the arguments correctly 92%.

Traditional catalogs were measured by coverage. An agentic catalog has to be measured by outcomes: can an agent answer a real question correctly, explain why, and know what it doesn’t know? Waiting for real users to surface enough varied questions makes that loop painfully slow — so we built Rex, a simulation engine that encodes five years of persona research (800+ customer interviews, tens of thousands of real queries) into executable tests.

Rex generates personas grounded in how people actually ask, runs live multi-turn simulations against the real instance, and scores every turn across seven dimensions — the most important being why something failed. When a conversation falters, Rex diagnoses the cause and sorts it: a metadata gap (a missing description, owner, or README) or an AI gap (retrieval or reasoning).

Simple context gaps are closed automatically by Context Agents, without a human triaging each one; the complex ones are routed to the right owners. The human moves from in the loop to on the loop — reviewing what was closed rather than doing the closing. Every run enriches the context lakehouse, so the next run is better.

This is the learning problem — the third one — finally answered. It is the compounding loop the old catalog never had: agents generate context, agents and people use it, real and simulated interactions expose the gaps, the right layer gets fixed, and the next answer improves.

Put the pieces together and the shape is simple:

1. Context Agents solve creation, writing context from the evidence in your estate.
2. The Context Lakehouse turns your catalog into AI-ready data and exposes it as a vectorized, graph-traversable, queryable data structure.
3. MCP connects that foundation to the harnesses humans and agents work in.
4. Rex solves learning, turning every use into a signal that makes the next answers better.

One governed foundation, three problems finally answered, and a loop that compounds.

The agentic data catalog is the data and metadata foundation a broader context layer is built on — not the whole thing. The catalog knows the data estate: what exists, what it means, how it moves, who owns it, what policies apply, whether it can be trusted.

But an enterprise agent needs more than data context. It needs to know how work gets done — which tools, which sequence, which constraints, when to ask for approval, what a good outcome looks like. That is agentic context. I think about it as three stages of maturity:

The first two stages are where the agentic data catalog lives. The third extends into the full context layer — and into a future Agent Registry, where the old cataloging problems (discovery, ownership, versioning, trust, evaluation, reuse) reappear for the capabilities of agents rather than data assets. That registry is part of the broader context layer; the agentic data catalog is one of its foundations.

What is dying is not metadata, governance, or lineage — it is the idea that a catalog is a destination built for people to populate and browse. What comes next is the agentic data catalog: Context Agents solve creation, the context lakehouse and MCP solve consumption, and Rex closes the loop on learning — the AI-ready foundation the enterprise’s AI is built on.