How you structure context determines whether an AI agent reasons over the right facts or a pile of undifferentiated text. Get the four inputs conflated into one block, and the agent can no longer tell a standing instruction from a fact it retrieved five minutes ago. Platforms implementing dedicated structure today include Atlan, LangChain, Mem0, and LlamaIndex, each solving a different layer of the same problem. Gartner’s 2026 D&A Summit put a number on what happens when the schema has no semantic foundation underneath it: 60% of agentic analytics projects relying solely on MCP will fail by 2028 without semantic foundations. The schema is necessary. It is not sufficient.
| What It Is | The four-part schema (system, retrieved, memory, tool context) that organizes what an agent’s runtime sees at each step |
|---|---|
| Key Benefit | A consistent structure lets agents distinguish standing instructions from situational facts, cutting instruction drift and retrieval noise |
| Best For | Teams building production agents on LangGraph, MCP, or custom orchestration who need a repeatable context contract |
| Core Components | System context, retrieved context, memory context (short-term and long-term), tool context |
| Common Failure Mode | A well-structured schema fed by ungoverned data: current format, stale or uncertified content |
What is agent context structure?
Permalink to “What is agent context structure?”Agent context structure is the schema that organizes everything an AI agent needs to reason and act at a given step, separated into distinct, typed fields rather than a single undifferentiated block of text. It is not the same as context engineering, which is the broader discipline of deciding what content fills that schema and when. Structure is the container. Engineering is the practice of filling it correctly, refreshing it, and trimming it as a task unfolds.
The distinction matters because most teams reach for prompt-level fixes when their actual problem is architectural. A survey of context engineering for large language models frames the discipline around four foundational components: representation and encoding (plain text versus structured schemas like JSON, embeddings versus symbolic form), selection and filtering (relevance scoring to decide what enters the window), compression and distillation (reducing size without losing meaning), and temporal management (how context evolves across a multi-step run). Structure is where representation and encoding get decided; the other three components operate on top of whatever structure you chose.
LangChain’s own documentation for context engineering in agents draws a parallel line: model context (transient, reset every call: system prompts, message history, available tools, response schemas), tool context (persistent, read from and written to state, store, or runtime), and life-cycle context (middleware hooks between steps for summarization, guardrails, and logging). Whether you use LangChain’s taxonomy or a simpler four-part model, the principle is the same: context that is not typed and separated by function gets treated as equally authoritative by the model, and that is where instruction drift and hallucinated confidence both start.
Four schema fields organize what an agent sees. The governed substrate underneath determines whether what fills them is true.
How context structuring works
Permalink to “How context structuring works”Context structuring works by separating an agent’s inputs into typed fields, ordering them by priority within a limited token budget, and refreshing each field on a schedule that matches how quickly its underlying data changes. Get the separation wrong and an agent cannot tell a standing rule from a retrieved fact. Get the refresh cadence wrong and the agent acts confidently on data that is already out of date.
System context: what persists across the whole session
Permalink to “System context: what persists across the whole session”System context holds the agent’s role, its objective for the current task, hard constraints, and the tools it can call. It should stay stable across a session and separate from anything retrieved at run time. When system instructions and retrieved documents sit in the same undifferentiated block, models weight them similarly, which is a documented source of instruction drift in long multi-turn runs. This is also where agent primitives like role and goal definitions live before any task-specific content arrives.
Retrieved context: what gets pulled in and how it is labeled
Permalink to “Retrieved context: what gets pulled in and how it is labeled”Retrieved context is the documents, records, and metadata an agent fetches through retrieval augmented generation or a direct query to a data source. Structuring this layer well means every retrieved item carries its own metadata: source, timestamp, certification status, and a relevance score, not just raw text. Without that metadata attached, an agent has no way to distinguish a certified, current record from a deprecated one; both simply look like “content in the window.” This is the layer where a context graph does work a vector index alone cannot, because it carries relationships and trust signals along with the retrieved fact.
Memory context: short-term state versus long-term facts
Permalink to “Memory context: short-term state versus long-term facts”Memory context splits into working memory, scoped to the current session and discarded when it ends, and long-term memory, persisted across sessions in a store the agent queries explicitly. Types of AI agent memory map roughly onto this split: episodic and working memory behave like session state, while semantic and procedural memory behave like the long-term store. Treating both as one undifferentiated blob is a common mistake; it means an agent either forgets things it should retain, or drags stale session detail into contexts where it no longer applies. Getting this split right is also how teams avoid agents forgetting things they should have retained at exactly the wrong moment.
Tool context: structured inputs and structured outputs
Permalink to “Tool context: structured inputs and structured outputs”Tool context is the structured data an agent sends to and receives from function calls, APIs, or an MCP Server. This is where format matters most literally: a mature implementation puts schema validators at the MCP server level, pre-processing and simplifying structured data like tables or metadata records into a consistent shape before it ever reaches the model’s linguistic context. An agent that receives raw, inconsistently shaped tool output has to parse format on every call, which burns tokens and introduces more surface area for error than one that receives a validated, typed response every time.
How structure changes with schema maturity:
| Aspect | No structure (single text blob) | Basic structure (labeled sections) | Full schema (typed, governed fields) |
|---|---|---|---|
| Agent can distinguish instruction from fact | No | Partially | Yes |
| Retrieved content carries trust signals | No | No | Yes |
| Memory persists correctly across sessions | Rarely | Sometimes | Yes |
| Tool I/O is consistently typed | No | No | Yes |
| Failure mode | Instruction drift, hallucination | Inconsistent behavior across runs | Governed, auditable context |
Structure alone gets you consistency of form. It does not get you correctness of content, that depends entirely on what’s feeding the schema, which is why the next section matters more than the schema itself.
Why structure breaks down without a governed substrate
Permalink to “Why structure breaks down without a governed substrate”A perfectly designed context schema still produces bad agent behavior if the data filling it is wrong, stale, or contradicted elsewhere in the organization. This is the gap between how agent frameworks talk about context and what actually causes agents to fail in production.
The production data says the problem is quality, not architecture
Permalink to “The production data says the problem is quality, not architecture”According to LangChain’s 2026 State of Agent Engineering report, based on 1,340 responses collected between November 18 and December 2, 2025, 57.3% of respondents now have agents running in production, with hallucinations and output consistency cited as the biggest challenge to agent quality at large organizations. A third of respondents overall cited quality, not orchestration or model choice, as the primary blocker to moving agents into production. Separately, over 70% of production agents have adopted a graph or state-machine architecture rather than a simple linear prompt chain, which tells you teams have already solved the structural problem at the orchestration layer. The blocker that remains is what fills that structure.
MCP standardizes delivery; it does not validate what is delivered
Permalink to “MCP standardizes delivery; it does not validate what is delivered”MCP (Model Context Protocol) gives agents a consistent way to request structured data from external systems, which solves the format half of the problem. It does not tell the agent whether the table it just retrieved is the certified version or a deprecated one sitting in the same schema. Gartner’s D&A Summit 2026 made this explicit: agentic analytics projects relying on MCP alone, without a semantic foundation underneath, are the ones Gartner expects to fail by 2028. Structure without governance is a well-organized way to deliver the wrong answer with total confidence.
What “governed” adds to a context schema
Permalink to “What “governed” adds to a context schema”A governed field in a context schema carries four things a raw field does not: certification (has this been quality-checked for this use), lineage (where did it come from and what changed upstream), ownership (who is accountable for it), and a semantic definition (what does this field actually mean in business terms, consistently, across teams). Business context for AI is what turns a retrieved record into something an agent can reason over correctly rather than merely display. Without these four signals attached to the schema, an agent’s context is structurally sound and semantically unreliable at the same time, which is a more dangerous failure mode than an unstructured mess, because it looks trustworthy.
Why context structure matters more as agents scale
Permalink to “Why context structure matters more as agents scale”Three forces are pushing context structure from an engineering nicety to a hard production requirement: agents moving from single-task assistants to multi-step autonomous workflows, multiple agents sharing the same underlying data, and enterprises needing to audit exactly what an agent saw before it acted. Deloitte projects that 75% of enterprises plan to deploy agentic AI within two years, which means the schema decisions made now will be running at a scale most teams have not tested.
Use case 1: multi-step agents that reason across a long task
Permalink to “Use case 1: multi-step agents that reason across a long task”An agent researching a customer account, checking inventory, and drafting a follow-up email needs its context schema to survive many steps without the system instructions getting diluted by everything retrieved along the way. Structured separation between system and retrieved context is what keeps the agent’s original goal intact by step twelve instead of step two. This is the core problem agent architecture documentation addresses when it discusses state persistence across a run.
Use case 2: multiple agents reading the same context store
Permalink to “Use case 2: multiple agents reading the same context store”When several agents query the same memory store or context graph, an unstructured or partially governed schema creates a coordination failure: one agent acts on a record another agent already flagged as stale. Context management in multi-agent systems depends on every agent reading the same typed, governed schema rather than each maintaining its own private, inconsistent version of “what’s true.” Multi-agent memory silos are what happens when that shared schema does not exist.
Use case 3: auditing what an agent knew before it acted
Permalink to “Use case 3: auditing what an agent knew before it acted”In regulated workflows, teams need to reconstruct exactly what data an agent saw, when, and whether that data was certified at the time. A schema with typed, governed fields makes this traceable. An undifferentiated context blob does not. This is where structured context intersects directly with AI agent governance: you cannot audit a decision you cannot decompose into its inputs.
How to structure context for AI agents
Permalink to “How to structure context for AI agents”Structuring context for an agent follows five steps: define the schema fields, separate them by persistence and function, attach trust metadata to anything retrieved, set a refresh cadence per field, and validate the schema against real multi-step runs before shipping it.
Prerequisites before you start:
- [ ] A clear list of every data source, tool, and memory store your agent will touch
- [ ] Agreement on which fields are transient (reset per call) versus persistent (saved to state or a store)
- [ ] A metadata source (catalog, glossary, or context graph) that can supply certification, lineage, and ownership for retrieved fields
- [ ] A test suite of multi-step agent runs to validate the schema holds up past the first two or three turns
Step 1: Define the four schema fields explicitly
Write down what belongs in system context, retrieved context, memory context, and tool context for your specific agent. Do not let any of these default to “whatever fits in the prompt.” A schema you cannot name is a schema you cannot debug when it breaks.
Step 2: Separate by persistence, not just by topic
Mark each field as transient (reset every model call, like a system prompt) or persistent (saved to state or an external store, like long-term memory). LangChain’s context engineering documentation treats this persistence boundary as the primary design decision, more fundamental than which framework or vector store you choose.
Step 3: Attach trust metadata to every retrieved field
Every record entering the retrieved-context field should carry its source, last-updated timestamp, and certification status alongside the content itself. This is the step most teams skip, and it is the one Gartner’s MCP-failure prediction traces back to directly.
Step 4: Set a refresh cadence per field, not one cadence for the whole schema
System context might refresh once per session. Retrieved context might refresh per query. Long-term memory might refresh on a schedule tied to how often the underlying business fact changes. A single blanket refresh policy either wastes calls refreshing stable fields or lets fast-changing fields go stale.
Step 5: Validate against multi-step runs, not single-turn tests
Test the schema across the same multi-step workflows your production agents will run, not isolated single-prompt tests. Instruction drift and memory bleed only show up after several turns; a schema that looks clean in a one-shot test can still fail by step eight.
Common pitfalls:
- Treating the context window as one field: collapsing system, retrieved, memory, and tool context into a single block is the single most common structural mistake, and it is the direct cause of instruction drift.
- Retrieving content without trust metadata: a record with no certification status or timestamp is indistinguishable from a correct one until the agent has already acted on it.
- One refresh cadence for every field: stable instructions do not need per-call refreshes; fast-changing operational data cannot survive a session-long cache.
- Skipping multi-step validation: schemas that pass single-turn tests routinely fail once memory and retrieved context start interacting across turns.
Getting the schema right is the engineering half of the problem. The next section covers the governance half, which is where most schemas quietly fail.
How Atlan approaches context structure for AI agents
Permalink to “How Atlan approaches context structure for AI agents”Most teams building agents today have solved the schema problem: a system prompt, a retrieval pipeline, a memory store, and a tool-calling interface, often through LangGraph, Mem0, or a custom MCP implementation. What most of those schemas lack is a governed source feeding the retrieved-context and tool-context fields with data that carries certification, lineage, and a consistent business definition.
Atlan’s Enterprise Data Graph is the governed data substrate that sits underneath an agent’s context schema. When an agent’s retrieval layer queries a data asset through Atlan’s MCP server, the response is not just the raw record; it carries the asset’s certification status, its lineage back through upstream transformations, its owner, and its definition from the business glossary, all structured as typed metadata the agent’s context schema can consume directly. This is what separates a schema that is well-formed from a schema that is trustworthy: the same JSON shape either way, but one carries governance signals and one does not.
As a concrete example, when an upstream table is deprecated, Atlan’s lineage engine propagates that change to every downstream asset. An agent querying a deprecated table through a properly structured context schema receives a certification-failed field rather than the same stale record it retrieved yesterday. The schema does not change; what fills the schema does, and that is precisely the layer most agent stacks have no mechanism for. Active metadata as agent memory is what makes the governed substrate continuously current rather than a one-time snapshot bolted onto a retrieval pipeline.
Real stories from real customers: context structure in production
Permalink to “Real stories from real customers: context structure in production”"We're excited to build the future of AI governance with Atlan. All of the work that we did to get to a shared language at Workday can be leveraged by AI via Atlan's MCP server…as part of Atlan's AI Labs, we're co-building the semantic layer that AI needs with new constructs, like context products."
— Joe DosSantos, VP of Enterprise Data & Analytics, Workday
"Atlan is much more than a catalog of catalogs. It's more of a context operating system…Atlan enabled us to easily activate metadata for everything from discovery in the marketplace to AI governance to data quality to an MCP server delivering context to AI models."
— Sridher Arumugham, Chief Data & Analytics Officer, DigiKey
A well-structured schema is only as reliable as what fills it
Permalink to “A well-structured schema is only as reliable as what fills it”Structuring context for AI agents is a solved engineering problem: separate system, retrieved, memory, and tool context, respect the persistence boundary between transient and saved state, and attach trust metadata to anything pulled in at run time. Every major framework, from LangChain’s model and tool context split to MCP’s standardized delivery format, gives teams the pieces to build this schema correctly today.
What remains unsolved at most organizations is the layer underneath the schema. A third of teams cite quality, not architecture, as their primary blocker to production, and Gartner’s own prediction ties agent failure directly to the absence of a semantic foundation beneath MCP, not to how the schema is shaped. The organizations closing that gap are the ones treating governed metadata as a prerequisite for context structure, not an afterthought layered on once the schema is already live. Explore how context architecture for AI agents extends this schema question into full system design, and how context freshness keeps a well-structured schema from decaying the moment it ships.
FAQs about how to structure context for AI agents
Permalink to “FAQs about how to structure context for AI agents”- How do you structure context for AI agents?
Structuring context for AI agents means organizing four distinct inputs, system instructions, retrieved knowledge, memory, and tool outputs, into a schema the agent’s runtime reads reliably at every step. The schema defines what goes in the context window, in what order, and how it is refreshed. It only produces reliable answers if the data behind it is current, certified, and semantically defined.
- What is the difference between context structure and context engineering?
Context structure is the schema itself, the fields, layers, and format an agent’s context takes. Context engineering is the broader discipline of deciding what goes into that structure, when, and why, including selection, compression, and governance. Structure is the container; engineering is the practice of filling it correctly.
- What should an agent’s system context contain?
System context should contain the agent’s role, its objective for the current task, operating constraints, and available tools, kept separate from retrieved data and conversation history. Mixing system instructions with retrieved content makes it harder for the model to distinguish standing rules from situational facts, which is a common source of instruction drift in multi-turn agents.
- How should retrieved context be formatted for an agent?
Retrieved context should carry structured metadata alongside the raw content: source, timestamp, certification status, and a relevance score. Representation and encoding, choosing plain text, JSON, or symbolic form, is one of the four foundational components of context engineering. Uncertified or stale records should be flagged before they enter the window, not after the agent has already reasoned over them.
- How do you structure memory for AI agents?
Structure agent memory into short-term working memory, scoped to the current session, and long-term memory, persisted across sessions in a store the agent queries at run time. LangChain’s framework separates these by persistence: transient model context resets every call, while state and store objects persist and must be explicitly read and written by the agent’s tools.
- What is the difference between context engineering and prompt engineering?
Prompt engineering shapes a single text input sent to the model. Context engineering designs the full environment the model reasons in: what data it can see, what memory persists, what tools it can call, and how all of that is structured and governed across an entire agent run, not just one inference call.
- Why do agents fail even when their context schema is well-designed?
A well-designed schema still fails if the data filling it is wrong. Structure determines where information goes; governance determines whether that information is current, owned, and trustworthy. Quality, not orchestration, is the top blocker for a third of teams moving agents into production, and the underlying cause is most often context that was structured correctly but never governed.
- What is a context schema at the MCP server level?
An MCP server context schema is a validated, typed contract for what an agent can request and receive from a connected data source through the Model Context Protocol. It pre-processes and simplifies structured data, such as tables or metadata records, into a consistent format before that data enters the model’s linguistic context, so agents receive predictable fields rather than raw, inconsistent payloads.
- Does a bigger context window remove the need to structure context?
No. Larger context windows change how much can fit, not whether an agent can find and trust the right parts of it. Long, unstructured context windows still suffer from position bias and irrelevant-content dilution, which is why selection, ordering, and filtering remain necessary even as raw window size grows.
