A production AI agent stack has six layers: models and inference, framework and runtime, tools and protocols (MCP), memory, the context layer, and observability and governance. According to Gartner (2025), over 40% of agentic AI projects will be canceled by the end of 2027, and according to MemU (2026), roughly 65% of agent failures trace to context drift rather than raw model capability. The layer most teams leave off the diagram, the context layer, is the one that decides whether agents work. This guide gives you per-layer decision criteria plus three trade-off tables for the real forks: build vs buy, framework vs raw SDK, and platform-native vs cross-platform context. Context = Knowledge + Expertise + Norms. MCP is the wire, not the meaning. For the what and why of each layer, see the companion explainer on the AI agent stack; this is the how and decide reference.
Why the AI agent stack needs decision criteria, not another diagram
Permalink to “Why the AI agent stack needs decision criteria, not another diagram”Naming the layers is the easy part. Deciding what goes in each one is where teams get stuck. According to BCG (2025), about 60% of companies report minimal or no measurable value from AI despite heavy investment, and the gap is rarely the model. Search results are saturated with layer-count diagrams, six in one, seven in another, nine in a third, and the count tracks whatever the author sells.
The best explainers list the tools per layer thoroughly. What none makes the spine of the page is the criteria a team uses to choose, and several conflate per-agent memory or RAG with a governed context tier instead of naming context as its own decision. According to O’Reilly (2026), the model layer is commoditizing while the layers above it hold defensible value, so the decisions that matter most are not about which model to call but about what feeds it.
The tier that encodes what the business means is missing from every neutral diagram. That is why agents that demo well hallucinate in production: the stack everyone draws can execute and remember, but it has no shared place to learn what your business actually means. This page is for the AI engineers, data-platform leads, and architects deciding build vs buy.
Get the full picture in one brief
The AI Context Stack brief lays out how the layers fit together and where the context layer sits, in a format you can share with your team.
Get the AI Context Stack briefWhat are the six layers of an AI agent stack?
Permalink to “What are the six layers of an AI agent stack?”A production agent stack reduces cleanly to six layers, justified from first principles rather than vendor convenience. Top to bottom they form a chain: intelligence, execution, reach, recall, meaning, control. Each layer answers a need the one below cannot.
The six layers of a production AI agent stack, from raw intelligence to governed, contextualized agents. Image by Atlan.
| Layer | What it does | Example tools (2026) | Build vs buy default |
|---|---|---|---|
| Models and inference | Raw intelligence and serving | OpenAI, Anthropic, Llama, vLLM | Buy (commoditizing) |
| Framework and runtime | Orchestrates the agent loop | LangGraph, CrewAI, OpenAI Agents SDK, raw loop | Depends (see trade-off) |
| Tools and protocols (MCP) | Gives the agent reach into systems | MCP, A2A, function calling | Buy / standard |
| Memory | Working, episodic, semantic recall | Mem0, Zep, Letta, pgvector, Neo4j | Buy or build |
| Context layer | Encodes what the business means | Enterprise Data Graph, Context Lakehouse, MCP Server | Buy or build (decision below) |
| Observability and governance | Eval, tracing, guardrails, policy | Langfuse, OpenTelemetry, NeMo Guardrails | Buy |
A commoditization gradient runs through this table. According to O’Reilly (2026), the bottom layers commoditize as model differences shrink each quarter, while the top layers hold defensible value. Models commoditize; context compounds, which is why the row most teams skip is worth investing in. For each layer in depth, see the companion explainer on the AI agent stack.
How do you decide what goes in each layer?
Permalink to “How do you decide what goes in each layer?”This is the part every competitor skips. For each of the six layers, here is the specific question a team answers to choose what goes there. The table below turns each layer into one clear fork.
| Layer | The question you actually answer | Pick X when… | Pick Y when… |
|---|---|---|---|
| Models and inference | Hosted API or self-hosted? | API: you want speed and the latest models | Self-host: cost at scale or data residency rules |
| Framework and runtime | Loop or framework? | Raw loop: a model plus a few tools | Framework: graphy flows or multi-agent roles |
| Tools and protocols | Custom integration or MCP? | Custom: one bespoke system | MCP: portability across many tools and agents |
| Memory | Per-agent store or shared? | Per-agent: a prototype or single agent | Shared and governed: to avoid context sprawl |
| Context layer | Platform-native or cross-platform? | Native: a single cloud, a single platform | Cross-platform: multi-cloud, portable, governed |
| Observability and governance | Logs or evals? | Logs: pre-production debugging | Evals and guardrails: anything in production |
Each fork resolves to a concrete signal. You stay on hosted model APIs until token volume makes self-hosting cheaper or a data-residency rule forces the workload in-house. A single prototype agent is fine with its own memory, but once a second agent needs the same corrected fact, a shared layer wins.
The context-layer row is the one most teams get wrong. According to O’Reilly (2026), 89% of teams running production agents have observability but only 52% have evals, so most can see failures without preventing them. According to the Gartner Market Guide for Agentic Analytics (2026), 60% of agentic analytics projects relying solely on MCP will fail by 2028 without a consistent semantic layer. Platform-native context locks meaning inside one cloud; a cross-platform context layer keeps it portable and governed.
Build vs buy, framework vs raw SDK: which trade-offs actually matter?
Permalink to “Build vs buy, framework vs raw SDK: which trade-offs actually matter?”Three forks decide most of the cost and lock-in: build vs buy for every layer, framework vs raw SDK at the runtime layer, and platform-native vs cross-platform at the context layer.
| Dimension | Build | Buy | Best for |
|---|---|---|---|
| Time to value | Slow | Fast | Buy if you are shipping this quarter |
| Control | Full | Bounded | Build for novel core IP |
| Maintenance | You own it | Vendor owns it | Buy for commoditized layers |
| Cost curve | High upfront | Predictable | Build only where it differentiates |
Teams most often get the framework decision wrong by reaching for too much too early.
| Dimension | Raw model API + simple loop | Framework (LangGraph / CrewAI) | Choose when |
|---|---|---|---|
| Setup cost | Minimal | Higher | Raw for a model and a few tools |
| Flexibility | Total | Opinionated | Raw, then drop it once it fights you |
| Multi-agent / graphy flows | Manual | Native | Framework for genuine graph or role play |
| Debuggability | You see everything | Abstraction tax | Raw early, framework when state grows |
According to Paolo Perrone, author of “The AI Agents Stack (2026 Edition)” at O’Reilly: “Most teams pick too much framework. If your agent calls a model and a few tools, you don’t need LangGraph.” Most effort goes to observability, retries, cost tracking, and debugging, not the framework. If you do need one, see how teams handle multi-agent orchestration first.
The third fork decides where your business meaning lives.
| Dimension | Platform-native (Unity Catalog / Horizon) | Cross-platform context layer | Best for |
|---|---|---|---|
| Coverage | One platform’s data | All sources, one graph | Cross-platform if you are multi-cloud |
| Portability | Locked to the platform | Model- and platform-agnostic | Cross-platform to avoid lock-in |
| Context drift | Fragments across silos | One governed, versioned source | Cross-platform at 30+ agents |
| Ownership | Per-platform | Org-wide ontology | Cross-platform for one source of meaning |
Every cloud platform is racing to build a context layer: Databricks shipped Unity Catalog Business Semantics, Snowflake has Horizon, Google introduced the Cloud Knowledge Catalog. The real question is not which catalog but where your whole organization’s ontology lives. Platform-native context fragments meaning across silos; a cross-platform context layer unifies it.
Put a number on the gap
The Context Gap Calculator estimates how much fragmented context is costing your agents, and what closing it is worth.
Try the Context Gap CalculatorWhy do AI agents fail in production, and what does the stack get wrong?
Permalink to “Why do AI agents fail in production, and what does the stack get wrong?”Most agent failures are not model failures, and three stack mistakes cause the bulk of them. Each looks reasonable on a whiteboard and breaks once real agents run against real systems.
The first mistake is treating the layer count as the answer. Counts are arbitrary, ranging from six to nine depending on who is selling what. Decide your layers from first principles, and the right number falls out of the needs your agents actually have.
The second mistake is leaving out a named context layer. Folding context into per-agent RAG and memory creates the context sprawl and drift wall, where a correction made for one agent never reaches the others. According to MemU (2026), roughly 65% of agent failures trace to context drift, not model capability. For the full list of root causes, see why AI agents fail in production.
The third mistake is too much framework too early. Pedro Domingos, Professor of Computer Science (Emeritus) at the University of Washington, argues that “harness engineering is the new prompt engineering.” That is complementary, not contradictory, because the harness has to read business meaning from somewhere, and that somewhere is the context layer. An optimized loop with no governed context delivers wrong answers faster.
How Atlan’s Context Layer for AI fits the stack
Permalink to “How Atlan’s Context Layer for AI fits the stack”If the context-layer row pointed you toward a cross-platform, model-agnostic option, Atlan is one implementation of that choice. It provides the context layer in this stack, the tier that encodes what the business means and delivers it to any agent, regardless of model or cloud.
Without it, teams hand-assemble context for every agent: the definitions, the joins, the policy. According to enterprise buyers in Atlan’s research, the result is that “there’s kind of no universal context being used in these AI agents,” and context drifts the moment one agent’s correction fails to reach the next.
The Context Layer for AI changes this by unifying context into one Enterprise Data Graph. Context Agents auto-generate agent-ready context such as descriptions, metrics, and glossary links; the Context Engineering Studio bootstraps and simulates that context before production. The Atlan MCP Server then delivers governed context to any agent or tool, because MCP is the wire, not the meaning. As intelligence commoditizes, context compounds: P = f(I, C).
The results are first-party and measurable. According to Atlan AI Labs, enriching agents with governed business context improved AI SQL accuracy by 38% across 522 evaluations, and Workday saw a 5x improvement in AI response accuracy after grounding agents via Atlan’s MCP server. Once you have decided the stack, the next step is implementing the context layer.
Real stories from real customers: Context layers in production
Permalink to “Real stories from real customers: Context layers in production”"We're excited to build the future of AI governance with Atlan. All of the work that we did to get to a shared language at Workday can be leveraged by AI via Atlan's MCP server…as part of Atlan's AI Labs, we're co-building the semantic layer that AI needs with new constructs, like context products."
Joe DosSantos, VP of Enterprise Data & Analytics, Workday
"Atlan is much more than a catalog of catalogs. It's more of a context operating system…Atlan enabled us to easily activate metadata for everything from discovery in the marketplace to AI governance to data quality to an MCP server delivering context to AI models."
Sridher Arumugham, Chief Data & Analytics Officer, DigiKey
Where is your context maturity today?
The Context Maturity Assessment scores how ready your organization is to feed agents governed context, and what to fix first.
Take the AssessmentModels commoditize, context compounds: build the layer that decides the stack
Permalink to “Models commoditize, context compounds: build the layer that decides the stack”The six-layer view is useful, but the decisions inside each layer separate agents that ship from agents that get rebuilt. Buy the commoditizing layers, reach for a framework only when graphy flows or multiple roles demand it, and keep observability honest with evals. The single decision that compounds is the one most diagrams omit: naming a context layer and choosing a cross-platform, governed one over context fragmented across clouds.
As model intelligence becomes table stakes, the proprietary asset is the context that tells agents what your business means. Build the stack so the harness, the memory, and the tools all read from one governed source.
FAQs about building the AI agent tech stack
Permalink to “FAQs about building the AI agent tech stack”1. How many layers does an AI agent stack need?
Permalink to “1. How many layers does an AI agent stack need?”Six layers cover a production stack from first principles: models, framework, tools, memory, context, and observability. Practitioners often run about four in a prototype, then add the rest as they move to production.
2. What is the context layer in an AI agent stack?
Permalink to “2. What is the context layer in an AI agent stack?”The context layer is the tier that encodes what the business means: Context equals Knowledge plus Expertise plus Norms. It is distinct from RAG and per-agent memory because it is shared, governed, and versioned across every agent.
3. Context layer vs data layer: what is the difference?
Permalink to “3. Context layer vs data layer: what is the difference?”The data layer stores and serves the data itself. The context layer sits above it and encodes meaning, ownership, and policy, so an agent knows not just the values but what they mean and what is allowed.
4. Do I need a framework like LangGraph or can I just call the model API?
Permalink to “4. Do I need a framework like LangGraph or can I just call the model API?”Use a raw loop when your agent calls a model and a few tools. Reach for a framework like LangGraph or CrewAI when you need graphy control flow or multiple distinct agent roles.
5. What is the difference between an agent and a workflow with a few LLM calls?
Permalink to “5. What is the difference between an agent and a workflow with a few LLM calls?”An agent loops and decides its next action based on results, so its path is not fixed in advance. A workflow runs a predetermined sequence of steps with no autonomous decision about what to do next.
6. Should I build or buy my AI agent infrastructure?
Permalink to “6. Should I build or buy my AI agent infrastructure?”Buy the commoditizing layers such as models, protocols, and observability, where vendors maintain parity. Build only where the capability is core IP, and treat the context layer as the place that compounds.
7. Why do AI agents fail in production?
Permalink to “7. Why do AI agents fail in production?”Most agent failures are context failures, not model failures. Roughly 65% trace to context drift during multi-step reasoning, which is why a shared context layer matters more than a stronger model.
Sources
Permalink to “Sources”- The AI Agents Stack (2026 Edition), O’Reilly. https://www.oreilly.com/radar/the-ai-agents-stack-2026-edition/
- Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, Gartner. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
- Are You Generating Value from AI? The Widening Gap, BCG. https://www.bcg.com/publications/2025/are-you-generating-value-from-ai-the-widening-gap
- Cube Recognized in the 2026 Gartner Market Guide for Agentic Analytics, Cube. https://cube.dev/blog/cube-recognized-in-the-2026-gartner-r-market-guide-for-agentic-analytics
- Context Drift Causes 65% of Enterprise AI Agent Failures, MemU. https://memu.pro/blog/ai-context-drift-enterprise-agent-memory
- AI Agents Are Scaling Faster Than Their Guardrails, Deloitte. https://www.deloitte.com/us/en/insights/topics/emerging-technologies/ai-agents-scaling-faster.html
- Harness Engineering Is the New Prompt Engineering, Pedro Domingos. https://x.com/pmddomingos/status/2067664169105830297
