Use context simulation for high-volume, multi-turn, and repeatable agent testing. Use manual QA for exploratory, subjective, or high-risk cases that require human judgment. Atlan’s Context Layer for AI grounds both: its Context Engineering Studio builds and certifies the simulated scenarios, so manual reviewers spend their time on the judgment calls simulation can’t make.
The split is easiest to understand through these five testing needs:
-
Scale and system behavior: Simulation tests many user paths, tool failures, and system conditions.
-
Multi-turn behavior: Simulation checks memory and state across longer conversations.
-
Regression testing: Simulation reruns past scenarios after prompt, model, tool, or API changes.
-
Exploration and experience: Manual QA finds unusual edge cases and judges tone, empathy, and usability.
-
Risk and sign-off: Manual QA handles ambiguous policies and high-stakes decisions requiring accountable review.
Below, we explain why manual QA loses coverage, how context simulation works, when each method fits, where human review remains essential, and how Atlan supports a combined workflow.
Why does manual QA run out of runway first?
Permalink to “Why does manual QA run out of runway first?”Manual QA runs out of runway because testers must decide which scenarios to write and run, while an agent’s possible paths keep multiplying. The limiting factor is how many combinations a human team can cover.
Consider a request such as “What was Q4 revenue?” One test may confirm the expected answer. Production users can ask the same question with different regional definitions, fiscal periods, permissions, follow-ups, and data conditions. Each variation can change the context retrieved, the tools called, and the final answer.
One global lifestyle brand spent roughly six weeks manually testing a customer-service analytics agent and still could not define when testing was complete. The team could keep adding cases, but the open-ended question space made an exhaustive hand-written test set unrealistic. That is how teams become trapped in AI agent POC testing hell.
Agent-evaluation research supports the wider mechanism. A 2026 revision of the academic survey on evaluating LLM-based agents says evaluation must cover sequential decisions and interactions in dynamic environments. It also identifies fine-grained, scalable evaluation as an open need and describes reliability across independent runs. A successful manual test therefore confirms one path under one set of conditions, not consistent behavior across varied paths.
Manual QA also serves a different purpose from other testing layers. Testing context quality for AI agents asks whether the underlying definitions, relationships, and rules are fit for use. Testing the agent harness checks components such as tools, memory, orchestration, and security. Decision traces help explain why a tested path failed. None of these methods automatically creates the missing scenarios.
Manual QA remains valuable for expert judgment. It becomes insufficient when it must also carry the full burden of scenario coverage. The next question is how context simulation expands that coverage without generating tests from guesswork.
What is context simulation, and what is it not?
Permalink to “What is context simulation, and what is it not?”Context simulation uses an LLM to generate and run many agent test scenarios from signals that reflect real business use. These signals can include BI dashboard questions, SQL history, known answers, and production traces. The result is a repeatable evaluation set covering more variations than a team could write manually.
Each source contributes different evidence. Dashboards contain metrics, filters, joins, and questions that people already use. SQL history shows which assets and query patterns recur. Production traces can reveal actual phrasing, follow-up behavior, and failures captured after deployment. User roles and access-control rules add another important source of variation.
A useful test scenario retains a clear connection to its source and an expected answer or scoring rule. Without that connection, an LLM may generate plausible questions that miss the organization’s fiscal calendar, metric definitions, regional exceptions, or policies. Without grounding, the LLM may produce a variety of questions, but there is no evidence that those questions represent real-world use. This is the same starting-point problem seen in context bootstrapping.
Generated scenarios also do not automatically behave like real users. A 2026 paper on goal alignment in LLM-based user simulators found that simulators can struggle to maintain assigned goals and behavioral constraints across multi-turn conversations. A separate 2025 survey of agent evaluation and benchmarking identifies realistic, scalable evaluation as an open research direction. Synthetic users therefore need constraints, source grounding, and checks of their own.
Context simulation does not replace adjacent testing methods. RAG evaluation measures retrieval and answer quality. AI observability explains what happened during a run. Context simulation supplies the scenarios across which those checks operate.
The evaluation set must also change with the system. Context drift detection can identify when definitions or source conditions have changed, while AI agent monitoring can surface production failures worth adding to the next test run. Context simulation is most useful as a continuing coverage process, not a one-time batch of synthetic questions.
How do scenario volume and failure stakes determine the method?
Permalink to “How do scenario volume and failure stakes determine the method?”Scenario volume determines how much testing must be automated. Failure stakes determine where human judgment and approval must remain. Teams need to assess both because a workflow can be high-volume, high-stakes, or both.
The four combinations lead to different testing mixes:
| Scenario volume | Failure stakes | Recommended testing mix | Why it fits |
|---|---|---|---|
| Low | Low | Manual QA with a small regression set | The limited number of stable cases is practical to review and repeat. |
| High | Low | Context simulation | Automated scenarios provide broad coverage without requiring a person to approve every result. |
| Low | High | Expert-led manual QA | A domain or risk owner can examine each rare case, document the evidence, and approve the outcome. |
| High | High | Context simulation plus human certification | Simulation tests the full range of scenarios, while people review high-impact failures and approve release criteria. |
Scenario volume is not simply the number of prompts in a test file. It is the number of meaningful combinations the agent may encounter across user roles, permissions, business domains, business definitions, context updates, tools, conversation turns, and system conditions.
For example, “Which orders are likely to miss their delivery date?” becomes several different scenarios when the region, shipping provider, delivery-status definition, user permissions, or follow-up question changes. Adding tool calls and agent handoffs introduces further paths. This is why scaling from one agent to many also expands the evaluation surface.
Failure stakes depend on three questions:
- What is the consequence of an incorrect result?
- Can the result be reversed?
- How quickly will someone detect the error?
An inaccurate internal search result may be easy to correct. An incorrect financial action, eligibility decision, or policy interpretation may require documented expert review before release.
A rare scenario does not become low-risk simply because it occurs infrequently. Likewise, a high-volume scenario does not require a person to inspect every run when its failures are reversible and easy to detect. The matrix separates the need for coverage from the need for approval.
The testing mix should change as the agent changes. A workflow can move toward higher volume as adoption grows, more data sources are connected, or additional agents and tools are introduced. It can move toward higher stakes when the agent gains permission to take actions rather than only provide information. Reassessing both helps prevent the limited test coverage that contributes to AI agents failing in production.
The high-stakes side of the matrix needs a closer look because some decisions should remain with qualified human reviewers.
When is manual QA still the right call?
Permalink to “When is manual QA still the right call?”Manual QA is still the right call when passing or failing a scenario depends on accountable human judgment rather than a stable expected answer. Simulation can generate and run these scenarios, but a qualified reviewer must interpret the result and decide what is acceptable.
Human review is most valuable in the following situations:
-
Ambiguous business definitions: A domain owner must decide which valid definition applies when business units calculate the same metric differently.
-
Subjective response quality: A reviewer must judge tone, clarity, empathy, or usefulness when several answers could be factually correct.
-
Unfamiliar and adversarial behavior: A tester explores unexpected instructions, prompt injection attempts, and unusual combinations that the existing evaluation set may not cover.
-
High-impact decisions: A risk, legal, or domain expert must review scenarios involving financial actions, eligibility, safety, privacy, or other difficult-to-reverse outcomes.
-
Release certification: An accountable owner must examine the evidence, accept known limitations, and approve the conditions under which the agent can be deployed.
Regulated testing also requires more than generating a large scenario set. Article 10 of the EU AI Act sets out data-governance practices and quality criteria for training, validation, and testing datasets used by certain high-risk AI systems. These datasets must be relevant and sufficiently representative for their intended purpose. Synthetic scenarios may contribute to testing, but their volume alone does not establish that the underlying data or evaluation process meets those requirements.
The NIST AI Risk Management Framework is voluntary and places trustworthiness considerations across the design, development, use, and evaluation of AI systems. For teams testing agents, this supports a defined review process in which people, responsibilities, evidence, and approval thresholds are established before release.
Manual review should therefore have a clear boundary. Teams should specify who reviews each high-stakes scenario, what evidence that person receives, what qualifies as a pass, and where the decision is recorded. Evaluation criteria for a context layer can provide a consistent review rubric, while bounded context spaces help keep domain-specific definitions and policies separate.
Once these human-only decisions are identified, the remaining coverage can be automated. The next step is generating that broader evaluation suite from real business signals instead of writing every test scenario by hand.
How does Atlan build a simulated evaluation suite from BI dashboards and SQL?
Permalink to “How does Atlan build a simulated evaluation suite from BI dashboards and SQL?”Atlan combines Context Engineering Studio with the Context Lakehouse to turn existing business context into a repeatable evaluation suite. Context Engineering Studio handles the build, test, review, and certification workflow. The Context Lakehouse stores and delivers the resulting context across agents and tools.
The workflow has five stages:
| Stage | What Atlan does | Why it matters |
|---|---|---|
| Scope | The team defines the business domain, intended users, relevant assets, and deployment target. | A clear boundary prevents unrelated definitions and data from entering the evaluation. |
| Bootstrap | Context Engineering Studio reads catalog context, BI lineage, semantic sources, dashboards, and query history. | The test set begins with observed business activity rather than an LLM’s guess. |
| Generate | The Studio creates representative questions from the repository’s assets, metrics, queries, and domain scope. | Teams gain broader coverage without writing every test scenario manually. |
| Simulate and review | The questions run against the target agent, producing results that expose missing definitions, filters, joins, synonyms, or relationships. | Domain experts can review failures, resolve ambiguity, and decide whether the context is ready. |
| Certify and deliver | The approved context and evaluation configuration are versioned in a Context Repo and made available to agents through the Context Lakehouse. | Multiple agents can use the same certified context instead of rebuilding it for each deployment. |
The grounding comes from Atlan’s Enterprise Data Graph, which connects business definitions with technical assets, lineage, ownership, usage, quality signals, and policies. These relationships help Context Engineering Studio identify which assets and questions belong in a particular evaluation set.
The Context Lakehouse provides the storage and delivery layer beneath this workflow. It combines a knowledge graph for connected meaning with Iceberg-native storage, vector search, and version history. Agents can access the context through interfaces such as MCP, agent-to-agent protocols, SQL, and APIs.
This separation is important. Context Engineering Studio determines whether an agent can answer representative business questions using an approved Context Repo. The Context Lakehouse makes that tested context reusable and traceable across agent environments. Production observations and corrections can then feed later simulations, allowing the evaluation suite to grow into a regression set.
Together, these capabilities support implementing an enterprise context layer for AI. They do not replace testing the model, security controls, memory, tool behavior, or other parts of the agent harness. Instead, they test and deliver the business context on which those systems depend.
Atlan therefore automates scenario coverage while preserving human certification. The remaining question is how teams divide those responsibilities before deployment and after the agent reaches production.
How do you run simulation and manual QA together?
Permalink to “How do you run simulation and manual QA together?”Context simulation and manual QA work best as a continuous loop, not competing methods. Simulation supplies the coverage needed for large and changing scenario sets. Manual QA supplies the judgment needed for ambiguous definitions, subjective quality, high-impact failures, and release approval.
Start with scenarios grounded in dashboards, SQL history, known answers, and production traces. Run them at scale, route uncertain or high-stakes results to the appropriate expert, and add approved corrections to the regression suite. As users, tools, data sources, and policies change, production findings should become inputs to the next evaluation cycle.
Atlan supports this workflow by using Context Engineering Studio to build and test evaluation suites, while the Context Lakehouse stores and delivers the approved context across agents.
AI Agent Context Readiness Checklist
Before you scale a simulation program, check whether your dashboards, SQL history, and production traces are actually ready to ground it.
Check Your ReadinessFrequently asked questions
Permalink to “Frequently asked questions”1. What is context simulation for AI agents?
Permalink to “1. What is context simulation for AI agents?”Context simulation is a testing method that generates and runs representative agent scenarios from observed business signals such as dashboards, SQL history, validated answers, and production traces. It tests how the agent behaves across variations in questions, context, tools, and user state. Its value depends on the quality and representativeness of the grounding signals and whether each scenario can be traced to its source.
2. Can synthetic test scenarios replace human QA testers entirely?
Permalink to “2. Can synthetic test scenarios replace human QA testers entirely?”No. Synthetic scenarios can automate repetitive coverage and expose variations that a hand-written test set may miss, but they cannot resolve ambiguous definitions, subjective criteria, or high-impact tradeoffs on their own. People remain responsible wherever organizational policy, applicable regulation, or risk controls require expert judgment and accountable approval.
3. How many test scenarios does an AI agent need before it is production-ready?
Permalink to “3. How many test scenarios does an AI agent need before it is production-ready?”There is no universal scenario count. Teams should set risk-based coverage and performance targets across important intents, user roles, context conditions, tool paths, failure modes, and high-stakes edge cases. Production readiness depends on meeting those targets across representative and repeated tests, addressing critical failures, and having monitoring and fallback plans, not on reaching a particular number.
4. What is the difference between AI agent evaluation and manual QA?
Permalink to “4. What is the difference between AI agent evaluation and manual QA?”AI agent evaluation is the broader discipline of measuring outcomes, trajectories, tool use, safety, reliability, and production behavior. Manual QA is one method within that discipline, based on people authoring or running cases and judging results. Context simulation is another method, designed to expand coverage and regression testing through generated, grounded scenarios.
5. When is manual testing better than simulation for AI agents?
Permalink to “5. When is manual testing better than simulation for AI agents?”Manual testing is most useful for exploratory and adversarial testing, subjective response quality, ambiguous definitions, and rare high-impact cases that require expert interpretation. It may also be sufficient for a small, stable test set, although repeatable regression cases should be automated when practical. Simulation can cover nearby variations, while a qualified reviewer makes the final acceptance decision where human judgment is required.
6. How do you generate realistic test scenarios from BI dashboards and SQL logs?
Permalink to “6. How do you generate realistic test scenarios from BI dashboards and SQL logs?”Start by extracting the measures, filters, joins, query patterns, and recurring assets found in trusted dashboards and SQL history. Pair each scenario with the required context, relevant user role, and a validated reference query, expected answer, or grading rule. Then generate variations around time periods, permissions, definitions, and follow-up questions while keeping each case traceable to its source signal.
7. What are the risks of testing an AI agent only with hand-written test cases?
Permalink to “7. What are the risks of testing an AI agent only with hand-written test cases?”Hand-written cases reflect what testers remember to ask, so they can overrepresent happy paths and familiar failures while missing rare, compound, or multi-turn behavior. They also become expensive to maintain as context, tools, permissions, and user behavior change. The largest risk is false confidence because a clean pass on a narrow set may say little about production coverage.
8. Does simulation testing work for regulated or high-stakes AI use cases?
Permalink to “8. Does simulation testing work for regulated or high-stakes AI use cases?”Simulation can support regulated or high-stakes testing by broadening coverage and surfacing rare combinations before release. It does not establish readiness or compliance by itself, so teams must validate the test set’s representativeness, document the methods and results, and meet the rules and controls that apply to the use case. Human review and approval should remain wherever the consequences, applicable requirements, or organizational risk thresholds call for accountable judgment.
9. Who should own AI agent test scenario generation: data teams or AI teams?
Permalink to “9. Who should own AI agent test scenario generation: data teams or AI teams?”One team should be accountable for the evaluation program, even though scenario generation requires several teams. AI or platform teams typically manage the evaluation system, execution environment, graders, and release process, while data teams provide source quality, provenance, semantic definitions, and usage signals. Domain and risk owners define or approve expected outcomes for ambiguous and high-impact cases.
Sources
Permalink to “Sources”- Survey on Evaluation of LLM-based Agents, arXiv (submitted 2025, revised 2026). https://arxiv.org/abs/2503.16416
- Evaluation and Benchmarking of LLM Agents: A Survey, arXiv (2025). https://arxiv.org/abs/2507.21504
- Goal Alignment in LLM-Based User Simulators for Conversational AI, arXiv (submitted 2025, revised 2026). https://arxiv.org/abs/2507.20152
- NIST AI Risk Management Framework, NIST. https://www.nist.gov/itl/ai-risk-management-framework
- EU AI Act, Article 10, European Commission AI Act Service Desk. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-10
- Evaluation types: Offline and online evaluation, LangSmith documentation. https://docs.langchain.com/langsmith/evaluation-types
