How to Build Context for LLMs in Enterprise: A 5-Layer Guide

Why most enterprise LLM context fails — and what the 5-layer architecture fixes

The standard approach to enterprise LLM context is: find a vector database, ingest relevant documents, build a RAG pipeline, deploy. It works in demos. It fails in production because it skips two layers that determine whether the other three are trustworthy.

The five-layer architecture addresses this sequentially:

Most teams start at layer 3. Layers 1 and 2 are not technical layers — they’re governance layers that require organizational work, not just engineering. That’s why they get skipped: they don’t ship as features, don’t appear in demos, and require investment before the LLM is even built.

The teams that skip layers 1-2 discover them retroactively when their LLM starts giving wrong answers in production — and the debugging is expensive because the problem is upstream of everything the engineering team built.

Prerequisites

Before starting, you need:

• A data catalog with certified assets, or a plan to build one — this is your source of truth for layer 1
• RBAC policies defined for your data assets — agents inherit these at layer 2
• A clear scope: which domain or use case are you building context for first?
• A delivery target: what LLM or agent framework will consume this context?

Starting narrow — one domain, one use case — is not a limitation. It’s the right architecture. Enterprise context systems built domain-by-domain are more maintainable and more governable than systems that try to ingest everything at once.

Layer 1: Identify and inventory your context sources

Before building retrieval infrastructure, you need to know what context exists, whether it’s worth building from, and what type of retrieval each source requires.

Step 1.1: Map relevant data assets

Start with your agent’s domain and use case. What data does it need to access? For a data discovery agent in the Finance domain, that might include: Finance-tagged catalog assets, business glossary terms in the Finance domain, lineage graphs for Finance pipelines, quality scores for key Finance tables, and documentation for Finance processes.

Resist the temptation to start broad. Ingesting your entire data estate into a context pipeline creates a maintenance and governance burden that grows faster than the utility.

Step 1.2: Assess quality

For each source type you identify, assess: Is this asset certified? Is ownership accurate? When was it last updated? What’s the staleness risk?

An uncertified table with missing ownership and documentation that hasn’t been updated in a year is not a good context source — it’s a source of confident wrong answers. Better to exclude it and flag it for governance work than to include it and let the LLM treat it as authoritative.

Step 1.3: Categorize context types

Different context types require different retrieval strategies later:

• Structured metadata (catalog attributes, quality scores, ownership) → structured query at layer 3
• Unstructured documentation (runbooks, policies, business context) → semantic search at layer 3
• Lineage (upstream/downstream relationships, transformation paths) → graph traversal at layer 3
• Business glossary (domain-specific term definitions) → exact match with semantic fallback

Output from Layer 1: A context source inventory — every relevant source, its quality score, its staleness risk, and its context type. This is the input to layer 2.

Common mistake: Skipping this step and going directly to ingestion. Ingesting everything without quality assessment means the retrieval layer returns results from bad sources at the same confidence as good ones.

Layer 2: Govern your context sources

Governance is the layer that determines whether the LLM can trust what it retrieves. Before a context source reaches an LLM, it needs certification, ownership, RBAC, and a staleness policy.

Step 2.1: Certify assets

Certification is a named owner’s declaration that an asset is trusted and production-grade. In Atlan, this is the certification workflow — a steward reviews an asset, validates its documentation and quality, and marks it certified. In other catalogs, the equivalent may be a “verified” flag, a quality tier, or an approval workflow.

For context purposes: certified assets should be retrieved and injected with their certification status explicitly. The LLM should know which assets are certified and which are provisional — and can modulate its confidence accordingly.

Step 2.2: Assign ownership

Every context source needs a named owner accountable for its accuracy. Ownership serves two purposes: it creates accountability (someone is responsible for keeping this context accurate), and it provides a contact for the LLM to surface when something needs human judgment.

Step 2.3: Apply RBAC

The RBAC policies that govern human access to data should also govern LLM and agent access to context. This is not a new policy — it’s extending existing policies to a new access pattern.

Agents and LLMs are assigned identities with scoped permissions. A Finance LLM gets the same catalog access a Finance analyst would have — not admin access, not cross-domain access. When RBAC isn’t applied to context access, the LLM surfaces sensitive data from domains the requesting user was never authorized to see.

Step 2.4: Set staleness policies

Each context type has a natural validity window. Business glossary terms change slowly — a weekly refresh may be sufficient. Pipeline quality scores change with every run — a near-real-time refresh is appropriate. Schema metadata falls somewhere in between.

Defining staleness policies at layer 2 determines what the layer 5 maintenance system actually does. Without explicit policies, the maintenance layer has no criteria for what to refresh and when.

Output from Layer 2: Governed context sources — certified, owned, access-controlled, with defined staleness policies. This is what gets ingested at layer 3.

Note: If your catalog isn’t governed yet, start there. Building retrieval on ungoverned data produces LLMs that give structured, confident, wrong answers — which are harder to catch than obviously wrong ones.

Layer 3: Structure your context for retrieval

With governed context sources identified, the structure layer builds the retrieval infrastructure and organizes retrieved context for LLM consumption.

Step 3.1: Choose retrieval strategy by context type

• Catalog metadata (asset descriptions, certification, ownership) → structured query via catalog API or MCP — filter by domain, certification status, asset type
• Documentation (runbooks, policies, governance docs) → semantic search — vector embeddings, cosine similarity ranking
• Lineage (upstream/downstream relationships) → graph traversal — hop-by-hop from asset to related nodes via lineage API or MCP tool
• Business glossary (domain-specific term definitions) → exact match on term name with semantic fallback for synonyms

Don’t apply semantic search to everything — it’s the wrong retrieval strategy for structured metadata and lineage. The LLM doesn’t benefit from “semantically similar” lineage nodes; it needs the actual lineage relationships.

Step 3.2: Include governance metadata in your schema

Every retrieved context item should carry its governance metadata alongside its content:

• Certification status (certified, provisional, deprecated)
• Owner name and contact
• Last updated timestamp
• Quality score (if applicable)
• Domain membership

This metadata goes into the context window with the content. The LLM uses it to calibrate confidence.

Step 3.3: Rank by relevance and governance signal

When multiple context items are relevant to a query, certified, current items should rank higher than provisional or stale ones. This is not just semantic relevance — it’s governance-weighted relevance. A certified, recently updated asset description is more valuable to the LLM than a similar but uncertified one.

Output from Layer 3: A queryable, structured context store with retrieval strategies per context type and governance metadata embedded.

Layer 4: Deliver context to the LLM

The delivery layer is how context gets from your store into the LLM’s context window. Three options:

MCP server (preferred for agentic use cases): The MCP (Model Context Protocol) server exposes your governed context as callable tools — get_asset_metadata(), get_upstream_lineage(), get_glossary_term(). Agents call these tools during reasoning without custom integration code. Adding a new context source means spinning up an MCP server; it doesn’t require changing agent code.

REST API or GraphQL: Traditional integration for custom agent implementations. More flexible, more integration work. Appropriate when MCP isn’t supported by your agent framework or when you need fine-grained control over the retrieval logic.

Embedded retrievers (LangChain, LlamaIndex): Context retrieval logic embedded directly in agent orchestration code. Good for initial development; harder to maintain at scale because retrieval logic is coupled to agent logic.

Assemble context for each request

For each agent query, the delivery layer:

1. Retrieves relevant context from each source (catalog, vector store, lineage graph, glossary)
2. Ranks retrieved items by relevance and governance signal
3. Compresses to fit the context window — prioritize certified, current, high-quality items
4. Injects governance metadata so the LLM knows which items are certified, provisional, stale, or access-restricted

Test with real queries. For 10 representative queries, inspect the assembled context: are the top items what you’d want the LLM to see? If not, the problem is usually in ranking or context type selection — go back to layer 3.

Output from Layer 4: Context delivery pipeline with governance signals embedded in every response.

Layer 5: Maintain context freshness

Freshness is the hardest layer, and the one that kills most enterprise LLM deployments in the months after launch. Context that was accurate at deployment drifts silently. Agents don’t know their context is stale — they continue operating on it with the same confidence they had when it was current.

The staleness pattern:

• Month 0: context is accurate, LLM outputs are reliable
• Month 3: some assets have changed, ownership has shifted, new pipelines have been added; context is partially stale
• Month 6: significant drift; LLM gives inconsistent answers; team blames the model

The fix is active maintenance built from day one — not retrofitted after the staleness cliff.

Step 5.1: Build change detection

Subscribe to change events from your catalog and data sources:

• Schema change events → refresh metadata context for affected assets
• Certification change events → update certification status in context pipeline
• Ownership change events → update owner records
• Pipeline completion events → refresh quality scores and lineage for affected assets

Most modern catalogs expose change webhooks or event streams. Atlan’s active metadata propagates changes in real time — catalog events trigger automatic context refresh without manual intervention.

Step 5.2: Implement TTL-based refresh

Even without explicit change events, context should refresh on a schedule. Use the staleness policies defined at layer 2:

• Business glossary terms: weekly refresh
• Asset metadata: daily refresh (or on catalog change event)
• Pipeline quality scores: refresh with each pipeline run
• Lineage graphs: refresh with each pipeline run or schema change

Step 5.3: Add staleness monitoring

Build visibility into context freshness: what percentage of your context store is past its TTL? Which domains have the highest staleness rates? Which context types haven’t been refreshed recently?

This monitoring makes staleness a visible metric, not a silent degradation. When context staleness rises, you investigate before LLM outputs degrade.

Step 5.4: Build feedback loops

When an LLM output is wrong, trace it back to the context that caused the error. Was the context stale? From an uncertified source? Missing lineage? The root cause of most LLM accuracy problems in production is a context quality issue. Feedback loops from LLM errors to context quality improvements make the system self-correcting over time.

Output from Layer 5: A living context layer that stays accurate as the data estate evolves, with monitoring that makes staleness visible before it causes production failures.

Putting the 5 layers together: the enterprise context architecture

The five layers are not a sequential build — they’re ongoing processes running in parallel, with governance cutting across all of them.

Governance (layer 2) is the horizontal that determines whether the other four work reliably.

Reference architecture for a data agent:

• Source: Atlan data catalog (certified assets, lineage, business glossary, quality scores)
• Govern: Atlan RBAC, certification workflow, active metadata staleness detection
• Structure: vector store for semantic search, Atlan API for structured metadata, lineage graph for traversal
• Deliver: Atlan MCP server — structured, governed context delivery as agent-callable tools
• Maintain: Atlan active metadata — real-time change detection triggers context refresh automatically

Common pitfalls — and how to avoid them

Real stories from real customers: Context infrastructure at DigiKey and Workday

"Atlan is much more than a catalog of catalogs. It's more of a context operating system…Atlan enabled us to easily activate metadata for everything from discovery in the marketplace to AI governance to data quality to an MCP server delivering context to AI models."

— Sridher Arumugham, Chief Data & Analytics Officer, DigiKey

Watch Now →

"We're excited to build the future of AI governance with Atlan. All of the work that we did to get to a shared language at Workday can be leveraged by AI via Atlan's MCP server…as part of Atlan's AI Labs, we're co-building the semantic layer that AI needs with new constructs, like context products."

— Joe DosSantos, VP of Enterprise Data & Analytics, Workday

Watch Now →

The first step isn’t retrieval — it’s governance

What DigiKey and Workday describe is the same five-layer architecture from two different starting points. DigiKey built the governed metadata foundation — the context operating system — and then activated it for AI delivery. Workday built the shared semantic layer first — the governed vocabulary and language — and then made it LLM-accessible via MCP.

Both teams spent significant time on layers 1 and 2 before they touched retrieval or delivery. That’s not a detour — it’s the architecture. The teams that start with a vector store and retroactively discover they need governance are spending that time anyway; they’re just spending it on debugging and remediation rather than building.

The enterprise context layer is the infrastructure that makes the five layers tractable at scale — governed catalog, active metadata, MCP delivery, and the maintenance infrastructure that keeps all of it current as the data estate evolves.

FAQs

1. What is the best way to give an LLM enterprise data?
The best approach is a governed context pipeline: identify relevant data sources in your catalog, certify them, structure them for retrieval (vector search for documents, structured query for metadata, graph traversal for lineage), deliver via MCP, and maintain freshness with change detection. Starting with unstructured RAG on ungoverned data produces fast but unreliable results.

2. How does RAG work for enterprise LLMs?
RAG (Retrieval-Augmented Generation) retrieves relevant documents from a store and injects them into the LLM’s context window. For enterprise use, RAG must be extended with governance: certification of source data, RBAC on retrieval, staleness management, and lineage context alongside document content.

3. Why does enterprise AI fail after launch?
Most enterprise AI failures trace back to context quality degradation. The data the LLM was given at launch becomes stale, ownership changes, certifications lapse, and new assets are added that aren’t in the context pipeline. Without active freshness management, context accuracy declines as the data estate evolves.

4. What is a context layer for LLMs?
A context layer is the governed data infrastructure that LLMs draw from — a combination of a certified data catalog, business glossary, lineage graph, vector store, and access control system. It’s the layer between the LLM and the raw data estate.

5. How do you implement RBAC for LLM context?
RBAC for LLM context means inheriting your existing data access policies at the context layer. Agents and LLMs are assigned identities with scoped permissions — they can only retrieve context they’re authorized to access. The MCP server or retrieval API enforces these permissions on every context request.

6. What is MCP for LLM context delivery?
MCP (Model Context Protocol) is a protocol standard that exposes context sources as structured tools that LLMs can call. Adding a new context source means spinning up an MCP server — it doesn’t require changing agent code. Atlan’s MCP server exposes governed catalog metadata, lineage, and glossary as agent-callable tools.

7. How do you keep LLM context fresh at enterprise scale?
Freshness requires change detection (subscribe to catalog change events), TTL policies per context type, staleness alerts injected into the context pipeline, and re-indexing triggers on pipeline completions, schema changes, and certification updates.

8. What is a context product?
A context product is a reusable, governed bundle of context for a specific domain — all relevant metadata, lineage, and glossary terms packaged and maintained by a domain team. Any LLM or agent serving that domain consumes the context product rather than querying raw catalog data directly.

Sources

1. DigiKey Context Readiness, Atlan Regovern
2. Workday Context as Culture, Atlan Regovern
3. Atlan AI Labs Ebook — The 5x Accuracy Factor
4. Model Context Protocol Specification, Anthropic
5. Enterprise Context Layer for AI, Atlan
6. AI Context Stack Guide, Atlan
7. How to Implement Enterprise Context Layer, Atlan
8. Context Engineering for AI Agents, Atlan
9. Bounded Context Spaces for AI Agents, Atlan
10. LangChain RAG Documentation