--- title: "Context Evals vs. Data Quality Testing for AI Agents" url: "https://atlan.com/know/ai-agent/ai-agent-evals/context-evals-vs-data-quality-testing/" description: "Compare context evals with data quality testing to see what each one catches, and learn how to build both into a reliable AI agent testing stack." author: "Karthik Pasupathy" author_role: "Contributing Writer — AI Context & Agents" published: "2026-08-31" updated: "2026-08-31T00:00:00.000Z" --- --- Context evals check whether an AI agent uses the right business meaning to produce the right answer. Data quality tests check whether the underlying data is complete, valid, and fresh. Atlan's Context Layer for AI connects both: Vera, Atlan's Data Quality agent, establishes source trust, while **Context Engineering Studio** tests and certifies whether an agent actually retrieves and applies that trusted data correctly. The split is easiest to see across five areas: * **Focus:** Context evals examine how an agent interprets context and completes a task; data quality testing examines datasets and pipeline outputs. * **Checks:** Context evals can expose wrong definitions, joins, permissions, tool calls, and actions; data tests catch nulls, duplicates, invalid values, broken relationships, and schema changes. * **Scoring:** Context evals may combine code checks, model graders, and human review; data tests usually return deterministic pass-or-fail results. * **Stage:** Context evals test agents before deployment and through continuous regression testing; data tests protect development and production data workflows. * **Relationship:** Data quality establishes reliable inputs; context evals test whether the agent uses those inputs correctly. Below, we examine where each approach fits, what it catches, and why high-quality data can still produce the wrong agent answer. --- ## Where do context evals and data quality testing fit in the AI stack? Data quality testing operates at the data and pipeline layer. Context evals operate at the agent layer, where retrieved data becomes business context, tool calls, decisions, and outcomes. Most enterprises need both because bad data can break AI models, while clean data cannot prove that agent behavior is correct. Atlan connects these layers through the [Context Layer for AI](https://atlan.com/context-layer/). Data quality signals help establish whether an asset is trustworthy, while [context testing for AI agents](https://atlan.com/know/ai-agent/ai-agent-evals/what-is-context-testing-for-ai-agents/) checks whether the agent retrieved and applied the correct definitions, relationships, and policies. For example, an agent can query fresh, complete regional revenue tables and still choose the wrong definition, join, permission, or tool argument. The data tests pass, but the context eval fails. According to RAND Corporation's 2024 interview research, 30 of 50 industry interviewees raised data quality problems on their own initiative, and 10 raised a lack of domain understanding as a separate failure cause. Although the study did not measure context-eval failures directly, it treats data readiness and domain knowledge as distinct concerns worth separate fixes. The same boundary applies to [context, data, and LLM observability](https://atlan.com/know/ai-agent/ai-agent-observability/context-observability-vs-data-observability-vs-llm-observability/): observability records what happened, while tests determine whether it met a standard. --- ## What does data quality testing actually check? Data quality testing checks whether source and transformed data meet defined rules across records, columns, tables, models, and pipeline outputs. The [dbt data test documentation](https://docs.getdbt.com/docs/build/data-tests) defines these tests as assertions about data resources: a SQL query returns records that violate the rule, and zero failing records means the test passes. dbt includes not_null, unique, accepted_values, and relationships, as well as custom tests and warning and failure thresholds. [Great Expectations](https://greatexpectations.io/expectations) uses a similar approach: an Expectation is a verifiable assertion, and a Suite groups expectations for one dataset. Across these tools and workflows, data quality programs commonly check the following conditions: * **Completeness:** Required fields and records are present. * **Uniqueness:** Keys and records do not contain prohibited duplicates. * **Validity:** Values match expected types, formats, ranges, categories, and schema rules. * **Referential integrity:** Keys and relationships remain consistent across datasets. * **Freshness:** Source and transformed data arrive within an agreed time window. These controls support [training data quality](https://atlan.com/know/how-to-ensure-llm-training-data-quality/), [data labeling](https://atlan.com/know/data-labeling-best-practices-llms/), [data quality in LLMs](https://atlan.com/know/data-quality-in-llms/), and [data quality inside an agent harness](https://atlan.com/know/data-quality-ai-agent-harnesses/). [Data observability for AI pipelines](https://atlan.com/know/ai-agent/data-for-ai/data-observability-for-ai-pipelines/) supports them by monitoring freshness, volume, distribution, schema, and lineage. These checks still cannot decide whether "active customer" means a recent login, a paid subscription, or a contract with positive recurring revenue. Choosing the definition requires a context eval. --- ## What do context evals actually check? Context evals check whether an agent uses the right context and follows the right process to complete a realistic task. Unlike data quality tests, the unit being tested is a task or trial, not a record, column, or table. Anthropic's 2026 guide to agent evals separates the transcript, including responses, tool calls, reasoning, and intermediate results, from the final outcome. For an enterprise data agent, a context eval can check five areas: * **Retrieved definitions:** Did the agent select the business definition that applies to the user's question? * **Joins and calculations:** Did it combine the correct datasets at the right level and apply the expected filters and metric logic? * **Permissions:** Did it respect the user's access rights and avoid retrieving or exposing restricted information? * **Tool calls:** Did it select the right tool and supply the correct arguments? * **Trajectory and outcome:** Did it follow a valid sequence of steps and produce the intended answer or action? Code-based graders can verify sources, tool arguments, permissions, and system states. Model-based graders and human reviewers can assess meaning or open-ended requirements. Because agent behavior varies, teams run multiple trials even when individual checks are deterministic. [Measuring agent trajectory](https://atlan.com/know/ai-agent/ai-agent-trajectory-evaluation/) reveals invalid steps, unsafe calls, and hidden retries. Preserving these [decision traces](https://atlan.com/know/what-are-decision-traces-for-ai-agents/) turns reviewed failures into regression tests. A 2025 [survey of LLM-based agent evaluation](https://arxiv.org/abs/2503.16416), revised in 2026, covers planning, tool use, safety, robustness, and evaluation frameworks. Unlike [LLM model benchmarks](https://atlan.com/know/llm-benchmarks-explained/), context evals test a single agent against company-specific tasks and rules. Teams can use [LLM evaluation frameworks](https://atlan.com/know/llm-evaluation-frameworks-compared/), [agent evaluation platforms](https://atlan.com/know/ai-agent/ai-agent-evals/best-ai-agent-evaluation-platforms/), or [alternative eval workflows](https://atlan.com/know/ai-agent/ai-agent-evals/braintrust-alternatives/), but realistic tasks and clear success criteria matter most. --- ## Where do clean data and correct agent answers diverge? Clean data and correct agent answers diverge when a dataset passes its tests, but the agent uses the wrong meaning, relationship, access rule, or source. A leading UK retail group encountered this gap when deploying an AI analyst. Setup took minutes, but correcting the business definitions and synonyms needed for accurate answers took much longer. Clean data can still produce the wrong result in several ways: * **Definition or semantic mismatch:** The agent selects a valid definition of revenue, customer, or churn, but not the one relevant to the user's domain or question. * **Join error:** The tables pass their individual checks, but the agent combines them using the wrong keys or at the wrong level of detail. * **Permission or authorization failure:** The data is valid, but the agent attempts to retrieve fields or records the user is not allowed to access. * **Retrieval failure:** The approved definition or source exists, but the agent does not retrieve it for the task. * **Stale context or context drift:** The data remains healthy while a definition, policy, relationship, or semantic model becomes outdated. A CRM SaaS company saw answer quality decline after adding semantic models and data sources, even though its data quality had not changed. The larger context pool made it harder to retrieve the right information. This is why [AI agent accuracy](https://atlan.com/know/ai-agent/ai-agent-accuracy/) is not only a model or data quality problem. [AI agent hallucinations](https://atlan.com/know/ai-agent-hallucination/), the reasons [LLMs hallucinate](https://atlan.com/know/llm-hallucinations/), and [RAG accuracy problems](https://atlan.com/know/rag-accuracy-problems/) can all involve missing, outdated, or misapplied context. Andrew Reiskind, Chief Data Officer at Mastercard, described the progression from "privacy by design" to "data by design" to "[context by design](https://atlan.com/regovern-2025-recap/)." Takashi Ueki, Head of Data and Enterprise Analytics at Elastic, similarly called for teams to scale "[data quality and context development](https://atlan.com/know/regovern-conversational-analytics-guide/)." [Detecting context drift](https://atlan.com/know/context-drift-detection/), therefore, means monitoring definitions, relationships, policies, and retrieval, as well as pipeline health. Otherwise, [silent context drift](https://atlan.com/know/context-drift-ai-agents/) can weaken answers while data tests remain green. These failures also show why a fixed set of test cases cannot represent every question, interpretation, and path an agent may encounter. --- ## Why can't static test suites cover an agent's real question space? Static test suites verify known scenarios, but agent inputs and action sequences vary. Teams must continually add new questions, paths, and failures to keep coverage representative. A global lifestyle brand spent roughly six weeks evaluating a customer service agent without reaching complete coverage. New wording, follow-ups, customer situations, and conversational paths kept revealing gaps. Three properties make agent evaluation an ongoing process: * **Variable questions:** Users can express the same intent with different wording, scope, follow-ups, and assumptions. * **Variable trajectories:** The same agent can retrieve different context or follow different tool paths across repeated attempts. * **Changing context:** Definitions, policies, available tools, data sources, and expected behavior can change after the original suite is written. A static suite remains a useful regression baseline, but broader coverage must learn from dashboards, verified queries, support tickets, production traces, and reviewed failures. For data-answering agents, dashboards reveal important questions and metrics, while verified SQL provides accepted sources, joins, filters, and calculations. A practical suite-building loop follows three steps: 1. **Collect realistic tasks:** Start with dashboards, verified queries, analyst requests, support issues, and production failures. Record the expected answer, context, permissions, and tools. 2. **Grade repeated trials:** Use deterministic checks for verifiable outcomes and calibrated rubrics for meaning, safety, or explanation quality. 3. **Expand the suite:** Turn valid failures, user corrections, and newly observed questions into regression cases. Anthropic recommends starting with 20 to 50 tasks drawn from real failures, then using repeated trials, reference solutions, multiple graders, transcript review, and continuous maintenance. The same principle applies when [evaluating RAG systems](https://atlan.com/know/how-to-evaluate-rag-systems-explained/) and [testing context quality](https://atlan.com/know/ai-agent/context-quality-testing-for-ai-agents/): teams must test [how much context is enough](https://atlan.com/know/how-much-context-is-enough/) and whether it meets their [context-layer evaluation criteria](https://atlan.com/know/ai-agent/context-layer/context-layer-evaluation-criteria/). A generated suite starts with real business activity and expands as the agent encounters new situations, with humans reviewing ambiguous or high-stakes cases. --- ## How do you build the eval layer on top of data quality tooling? Build the eval layer on top of existing data quality tests. The data layer asks whether inputs passed their checks. The agent layer asks whether those inputs were used correctly. A layered testing workflow can follow these steps: 1. **Validate and connect the data:** Keep schema, completeness, uniqueness, validity, relationship, freshness, and business-rule tests. Associate their results with assets, definitions, owners, certifications, and [training data lineage](https://atlan.com/know/training-data-lineage-for-llms/). 2. **Build and run realistic tasks:** Use dashboards, verified queries, analyst requests, and production failures. Record the expected outcome, sources, joins, permissions, [policy context](https://atlan.com/know/context-engineering-ai-governance/), and tools, then test retrieval, tool calls, trajectory, and outcome through the [agent harness](https://atlan.com/know/how-to-test-ai-agent-harness/). 3. **Route failures by layer:** Send failed datasets to the data pipeline. Treat wrong definitions, joins, sources, permissions, or [AI agent governance](https://atlan.com/know/ai-agent-governance/) decisions as context failures. 4. **Learn from production:** Use [context observability](https://atlan.com/know/ai-agent/ai-agent-observability/context-observability-for-ai-agents/) to turn corrections and failed requests into regression cases. Rerun affected evals after changes and use [A/B tests for LLM applications](https://atlan.com/know/ab-testing-llm-applications/) when comparing production versions. This connects to the [Enterprise Context Layer](https://atlan.com/know/context-layer-enterprise-ai/), which maps to the same steps: | Need in the testing lifecycle | Capability | Role in the eval layer | | :---- | :---- | :---- | | Establish source trust | [Connectors](https://atlan.com/connectors/) and [Vera, the Data Quality agent](https://atlan.com/data-quality-studio/) | Connected signals and asset scores indicate which data is complete, accurate, and up to date. | | Preserve meaning and provenance | [**Context Agents**](https://atlan.com/context-agents/) and [data lineage](https://atlan.com/data-lineage/) | Definitions, metrics, ownership, relationships, and column-level provenance support testing and root-cause analysis. | | Deliver governed context | [**Context Lakehouse**](https://atlan.com/context-lakehouse/) and the [**MCP Server**](https://atlan.com/mcp-server/) | Agents retrieve definitions, lineage, quality information, ownership, and policies through a shared layer. | This turns the [metadata layer beneath AI](https://atlan.com/know/metadata-layer-for-ai/) into machine-readable context, supporting a [governed RAG knowledge base](https://atlan.com/know/llm-knowledge-base-data-quality/). Release needs two passes: data meets its rules, and the agent uses the right context to complete the task. Teams structuring this can start from [implementing an enterprise context layer for AI](https://atlan.com/know/how-to-implement-enterprise-context-layer-for-ai/). --- ## How should teams combine data quality testing and context evals? Treat data quality testing and context evals as stacked controls, not competing approaches. Data tests verify that source and transformed data meet defined rules. Context evals verify that the agent retrieves the right definitions, uses the correct joins and permissions, calls tools correctly, and produces the intended outcome. Both layers must pass because clean data can still lead to an incorrect agent answer. Keep known test cases as a regression baseline, then expand the suite with real questions, verified queries, repeated trials, production failures, and user corrections. Connect these tests to quality signals, definitions, lineage, ownership, and policies so teams can identify where a failure began and prevent it from recurring. [Talk to Atlan](https://atlan.com/forms/talk-to-sales-contact/) to see how quality signals, governed context, evals, lineage, and production traces connect. --- ## Frequently asked questions ### 1. What is the difference between context evals and data quality testing? Data quality testing checks source and transformed data against defined rules. Context evals check whether an agent retrieves the right context and uses it correctly. They test different layers of the same AI system. ### 2. Can dbt tests or Great Expectations catch AI agent hallucinations? No, not directly. They can detect missing, invalid, or inconsistent data. They cannot determine whether an agent's answer is grounded, uses the correct definition, or follows a valid tool path. ### 3. Do you need both data quality checks and LLM or agent evals? Yes. Data quality checks verify the inputs, while agent evals verify how those inputs are interpreted and used. Neither replaces the other. ### 4. What is context drift, and how is it different from a data quality issue? Context drift occurs when definitions, relationships, policies, or retrieval behavior become outdated or inconsistent. A data quality issue affects the condition of a dataset or pipeline. Context can drift while the underlying data remains healthy. ### 5. How do you build an evaluation suite from dashboards and queries? Start with trusted dashboards and verified queries to identify real questions and expected answers. Record the required definitions, joins, filters, permissions, and tools for each task. Add reviewed production failures as new regression tests. ### 6. What is trajectory-level evaluation versus output-level evaluation? Output-level evaluation checks the final answer or outcome. Trajectory-level evaluation checks the retrieval steps, tool calls, and intermediate actions that produced it. Together, they can expose an invalid path that still produced the correct answer. ### 7. Where do data quality tests fit in the AI agent lifecycle? Data quality tests validate source and transformed data before and during agent operation. Context evals test the agent's behavior before release and after relevant changes. Together, they create a two-layer release gate. ### 8. What share of agent failures comes from data or context rather than the model? No credible universal percentage applies across every agent and use case. Teams should classify their own failures across data, retrieval, context, model, tool, policy, and environment causes. This produces a more defensible answer for their system. ### 9. How is context observability different from data observability? Data observability monitors the health and movement of datasets and pipelines. Context observability records which definitions, policies, sources, and relationships an agent used. Teams need both to determine where a failure began. ### 10. Does passing every data quality test guarantee an agent will answer correctly? No. It only shows that the data passed the rules being tested. The agent can still select the wrong source, apply the wrong definition, infer an invalid join, violate a permission, or misuse a tool. --- ## Sources 1. [The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, RAND Corporation, 2024](https://www.rand.org/pubs/research_reports/RRA2680-1.html) 2. [Survey on Evaluation of LLM-based Agents, Yehudai et al., 2025, revised 2026](https://arxiv.org/abs/2503.16416) 3. [Add data tests to your DAG, dbt Developer Hub](https://docs.getdbt.com/docs/build/data-tests) 4. [Expectation Gallery, Great Expectations](https://greatexpectations.io/expectations) 5. [Demystifying evals for AI agents, Anthropic Engineering, 2026](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)