Every data catalog we have built so far has carried the same assumption: somewhere on the other side of the screen, there is a human.
A human will document the table. A human will search for it. A human will read the description, inspect the lineage, ask a colleague what the metric really means, and decide whether the data can be trusted.
Even when we added AI to the experience, that assumption did not change. We made the search conversational. We drafted a description. We summarized a lineage graph. But the catalog was still designed by humans, maintained by humans, and consumed by humans.
That era is ending.
AI is becoming both a producer and a consumer of metadata. It can create the context that once took months of manual stewardship. And the fastest-growing audience for that context is no longer the analyst reading a catalog page. It is the agent trying to answer a question, investigate an anomaly, produce an analysis, or take an action.
For as long as it has existed, the catalog answered two questions: where is the data, and what does it mean. Discover, and understand. It is starting to answer a third. Now do the work.
This does not mean the data catalog disappears. It means the data catalog for humans becomes infrastructure for agents.
The three problems data catalogs never solved
Earlier, the catalog waited twice. Once for someone to create the context, then again for someone to come looking for it. Five years ago, in Data Catalog 3.0, I argued that metadata itself had become “big data”: too vast, dynamic, and interconnected for humans to maintain one field at a time. Query logs, lineage, and usage patterns could already help infer ownership, relationships, and context automatically. That thesis still assumed a human on the other side of the catalog.
The human was the supply chain: create, consume, and a learning loop that never closed | Source: Context and Chaos
Two were visible. The third nobody named.
Creation. Someone had to populate the catalog: descriptions, business terms, domains, owners, all kept current. At enterprise scale that was never possible, because the estate always changed faster than humans could document it. Now the gap widens. AI needs far more context than people ever did, and the same workforce that could not keep up with the old demand cannot possibly meet this one.
Consumption. Even when context existed, people had to remember to go looking for it: leave the workflow, open the catalog, translate a question into catalog language. We spent years reducing that friction with automation, lineage, Slack integrations, and natural-language search. None of it changed the model. Now it breaks in a new way. An agent reasoning inside Claude or Cursor has no interface into the catalog at all. A search box and a set of pages were built for human eyes.
Learning. This one was invisible because nobody expected it to be solved. The catalog never learned. It was measured by coverage, and coverage cannot tell you whether the context actually worked. When a definition was wrong, or missing, or right but misleading, nothing traced the bad answer back to the gap that caused it. Humans mostly stayed quiet, so the catalog never heard about its failures. Agents will not stay quiet.
The agentic catalog is what happens when all three finally get answered.
Phase one: solve the metadata supply problem
Humans cannot produce the volume of metadata AI requires, so the work moves to agents. Instead of starting from an empty field, an agent reads the evidence already sitting in the estate , lineage, SQL and query history, usage patterns, column metadata , and writes a grounded first draft. Humans set the standards, inspect the evidence, resolve ambiguity, and certify what can be trusted.
This is why “AI-powered catalog” is too small a frame. It sounds like an old catalog with a chatbot bolted on. The operating model itself changes: agents create and maintain first-pass metadata from evidence, humans move from writing every field to governing the system, and the catalog becomes a scalable supply of context AI can actually use.
Can agents really document a data estate?
We have been asking for more than two years. We shipped our first AI features in 2023, built a useful AI-powered catalog, and watched quality plateau at roughly 70%. The lesson was that this takes more than a better prompt. To be accurate, an agent needs rich signals, and we had stored all of them in systems built to serve a human interface rather than to be queried at the speed an agent needs. So in 2025 we rebuilt the foundation as a metadata lakehouse: the catalog’s knowledge landed as open Iceberg tables, queryable by anything that speaks SQL.
Then we built the agent harness on top of it, and rebuilt it more than once. Early this year, with stronger models and a far more rigorous context pipeline, it started working.
The first real test was a three-week cohort with 50 enterprises. Across them, agents generated 1.03 million descriptions and 15,000 READMEs. Those are our own program numbers and I would read them as such. The individual accounts are the more interesting part, and they kept returning to two surprises.
Agents recover knowledge nobody wrote down. Query-pattern analysis turned real usage into business-friendly questions, common joins, and important filters. Izabela Wilczyńska described what that felt like:
“The kind of knowledge that usually lives in one person’s head or a Slack thread suddenly became visible across thousands of assets.”
That is closer to knowledge synthesis than to documentation.
Query patterns turn undocumented usage into business-readable context | Source: Context and Chaos
Trust came from receipts, not fluency. The old objection to automated metadata is obvious: what happens when the machine invents a plausible but wrong definition? What changed skeptics’ minds was not scale but provenance. Cody Brees generated context across 103,515 assets and was convinced by the evidence behind each claim, not the number:
“The context traced back to something I could point at rather than arriving as an assertion.”
An agentic catalog cannot merely generate more prose. It must generate context with receipts.
The cohort gave us evidence that generation runs at enterprise scale. It did not prove the harder thing: whether the context that comes out is accurate, current, and useful at the moment another agent has to make a decision with it. A well-populated catalog does not help an agent that cannot find the right context, retrieve enough of it, and know what to do next.
The data steward becomes a context engineer
The natural fear is that if agents do the cataloging, the people who spent years doing it matter less. The opposite is more likely. When agents take on first-pass enrichment, human expertise moves up the stack: not filling empty fields, but deciding what good context looks like, which sources to trust, how to resolve conflicting definitions, testing agent output, and catching the exceptions where business judgment matters.
“It allows Data Governance professionals to spend more time validating information, improving context, and making decisions that require real knowledge of the business.” , Leonardo Vinicius Ruivo
The output is not perfect, and that is the point. Imperfection clarifies the human role rather than invalidating the model. Documentation stops being a periodic cleanup project and becomes a continuously enriched product. The people who spent years on metadata are unusually ready for this. They know enterprise meaning is messy, that a definition can be technically correct and still wrong for its audience, and that provenance and trust are what make data usable. They were context engineers before the industry had the name.
Phase two: AI becomes the consumer
Solving metadata supply is only half the problem. The catalog’s primary consumer may no longer be a person who searches and reads. It may be an agent inside ChatGPT, Claude, Copilot, an IDE, or a custom application. That agent does not need a better homepage. It needs a reliable way to retrieve the right context while it reasons, and for an agent reliable means four things at once: accurate, relevant, fast, and cheap in tokens. In The War of the Catalogs I argued that context creates the most value when it arrives where work is happening rather than requiring someone to visit another destination. In the last era that meant Slack, BI tools and notebooks. In this one it means the agent’s reasoning loop, and the most important consumer of the catalog may never see its homepage. Tokens are the currency, and a retrieval layer that wastes them degrades every step that follows.
This is the role of MCP. An MCP server exposes catalog capabilities as tools an AI client can call directly, under the same permissions that already govern the catalog. But MCP is only the interface, not the intelligence. The hard problem is context retrieval, and it has two dimensions.
An MCP server gives an agent access. Choosing the right door is a different problem | Source: Context and Chaos
Breadth: where should the agent look? A catalog holds millions of assets, and the same word can be a metric, a glossary term, a dashboard, or an entity. Search too broadly and the agent retrieves plausible but wrong context. Our first attempt at solving this was the obvious one, and it is what most of the industry did: wrap the API surface we already had and call it MCP. It failed. Agents do not think in endpoints. They think in tasks, and the tasks are more human than “metadata” suggests. People search by association, not by name. Someone asks what footfall looked like at a conference; nothing in the estate is called footfall, and something has to work out which metric that question means.
Depth: how much should it retrieve? Too little and the answer omits a column, a lineage edge, a policy. Too much and the response floods the context window, raises cost and latency, and buries what matters. We learned this the expensive way: a single search call of ours carried roughly 2,500 tokens of overhead before it returned anything useful. The fix is progressive disclosure , start with a small, ranked, high-confidence result, make the true scope visible, say honestly when something was truncated, then let the agent drill into exactly what it needs.
Both of those are design choices. Knowing whether they helped is a separate problem. Every tool we run is scored against an evaluation suite that asks whether the agent picked the right tool and called it with the right arguments. Those scores are good. But notice what they measure. They do not measure whether the person got their answer. That is the harder number, and it is the one that actually matters.
Phase three: the loop
Traditional catalogs were measured by coverage. An agentic catalog has to be measured by outcomes: can an agent answer a real question correctly, explain why, and know what it does not know?
Waiting for enough real users to surface enough varied questions makes that loop painfully slow. So you manufacture the questions instead. Generate representative personas, ask what those people would actually ask, record every retrieval decision, and attribute each failure to the part of the system that produced it. The point is not the simulated question. It is the trace of what happened next.
Traced, three failures that look identical from the outside come apart. Sometimes the context was missing. Sometimes it existed but the metric underneath it was never curated. Sometimes everything was there and the agent reached for the wrong thing. Each has a different owner, and without the trace teams blame the model and keep tuning the prompt.
This is the third problem, finally answered. It creates the compounding loop the old catalog never had: agents generate context, agents retrieve it, real and simulated use exposes the gaps, humans and agents fix the right layer, and the next answer is better. That loop is the whole of it. A catalog is not agentic because agents write into it, or because agents read from it. It is agentic when those two halves close on each other and the system gets better without being told to.
The loop the old catalog never had | Source: Context and Chaos
The agentic data catalog is not the context layer
Which makes it worth saying clearly what it is not. The agentic data catalog is the data and metadata foundation a broader context layer is built on, not the whole thing. The catalog knows the data estate: what exists, what it means, how it moves, who owns it, whether it can be trusted. But an enterprise agent needs more than data context. It needs to know how work gets done , which tools, which sequence, which constraints, when to ask for approval, what a good outcome looks like. That is agentic context.
| Stage | What it contains | The problem it solves |
|---|---|---|
| 1. AI-ready data | Metadata, descriptions, ownership, lineage, quality, usage, policy | Reliable context about the data estate. Agents solve the supply problem. |
| 2. Agent-ready data | Governed semantics, metrics, business entities, relationships, ontology | Consistent meaning, retrievable at runtime. |
| 3. Agentic context | Skills, tools, procedures, constraints, evaluations, operating knowledge | Teaches agents how to perform work, not merely how to understand data. |
The first two stages are where the agentic data catalog lives. The third extends into the full context layer, and eventually into a registry for agents themselves, where the old cataloging problems reappear for capabilities rather than data assets.
The next chapter belongs to the people who built the last one
For years, metadata and governance teams did essential work that was easy for everyone else to overlook. AI makes it visible, because the cost of missing context now appears immediately. The agent produces the wrong answer. It uses the wrong definition. It cannot explain its reasoning.
The solution is not a smarter model. It begins with a better metadata foundation, and not an improved one. A remade one.
The old catalog was built around a human sequence: document, search, inspect, decide. Agents break that sequence. They generate context continuously and consume it inside the act of reasoning, and most of them will never open a catalog page. That changes what the catalog has to optimize for. Coverage is no longer enough. Context needs evidence. Retrieval needs judgment. And every wrong answer needs a trace back to the definition, relationship, policy, or skill that was missing.
The new metadata foundation will increasingly be built and consumed by agents, but it cannot be entrusted to agents alone. It needs the people who understand how meaning is created, where it breaks, and why trust is earned.
The catalog for humans is dying. The people who built it are what make the next one work.
I’ve written the longer version of this, with the build log and the numbers: what we got wrong first, what the retrieval engineering actually cost, and how we measure whether any of it works. Read it here!
A few industry experts and leaders who have actually run this and are shaping the conversation around this are joining me live on September 10, 11am ET. An hour. Join us if it’s useful.
The Cats of Context & Chaos
The catalog has a new reader. | Source: Context and Chaos
About Context & Chaos
Context & Chaos isn’t just a newsletter. It’s shared community space where practitioners, builders, and thinkers come together to share stories, lessons, and ideas about what truly matters in the world of data and AI: context engineering, governance, architecture, discovery, and the human side of doing meaningful work.
Our goal is simple, to create a space that cuts through the noise and celebrates the people behind the amazing things that are happening in the data & AI domain.
Whether you’re solving messy problems, experimenting with AI, or figuring out how to make data more human, Context & Chaos is your place to learn, reflect, and connect.
Got something on your mind? We’d love to hear from you.
Share this article