In practice, an agent trajectory evaluation examines five parts: what tools were called, whether the arguments were correct, how many steps the task took, whether the agent’s plan held together, and how those actions compare to a reference path a domain expert would approve. Atlan’s Context Engineering Studio builds that reference by reading BI dashboards, SQL queries, and production traces, then Context Agents package the resulting logic into versioned Context Repos that any agent framework can query through the Context Lakehouse.
Quick Facts
| Concept | What it means |
|---|---|
| Ordered action log | The full sequence of reasoning, tool calls, and observations an agent produces during a task |
| Tool-call precision | Whether the agent selected the right tools with the right arguments at each step |
| Step efficiency | How many calls the agent made relative to the minimum needed, and whether it looped or retried |
| Plan adherence | Whether the agent followed a coherent strategy or stumbled into the answer by accident |
| Reference trajectory | The path a domain expert would have taken, used as the grading baseline |
Why isn’t a correct final answer enough?
Permalink to “Why isn’t a correct final answer enough?”Two agents can produce the same correct answer in very different ways. One makes three precise tool calls. The other makes forty calls, calls the wrong API twice, gets stuck in a retry loop, and reaches the answer by chance.
Output-only scoring treats both as successful.
But a correct answer does not make the second agent reliable. Change the question slightly, and its flawed process might fail the task. If you only evaluate the final output, you may not discover that risk until your agent is deployed in production.
According to the τ-bench researchers (2024), state-of-the-art function-calling agents succeed on fewer than 50% of realistic tool-use tasks. Output-only metrics can’t surface the failures that matter most: redundant tool calls, wrong tool selection, unsafe intermediate actions, or an agent that “succeeds” via a path no one would approve if they’d watched it happen.
Agent tracing tools record the path an agent took, including its tool calls, arguments, and intermediate results. But a trace alone cannot tell you whether the agent followed the right path for your business. That requires a reference built from real business context. As the Context Layer for AI, Atlan turns BI dashboards, SQL queries, and production traces into eval suites and reference trajectories that an agent’s path can be scored against.
Same final answer. Only trajectory evaluation catches the difference.
What is an agent trajectory, and what does it actually contain?
Permalink to “What is an agent trajectory, and what does it actually contain?”An agent trajectory is the step-by-step decision trace of an agent’s actions while completing a task. It captures the decisions an agent made, the tools it called, the inputs it sent, the results it received, and any changes it made along the way.
Think of it as an execution record, not a transcript of the agent’s hidden reasoning.
The exact record depends on the AI agent architecture and the agent harness that executes it, but a useful trajectory captures these AI agent primitives:
-
Task and constraints: what the agent was asked to do and the rules it had to follow
-
Visible plans or decisions: how the agent used its planning logic to choose its next action
-
Tool calls: which tools, APIs, or functions it selected as part of its AI agent tool use
-
Tool arguments: the parameters and data passed with each call
-
Intermediate results: what each tool returned and how the agent responded
-
State changes: any records created, updated, deleted, or sent to another system
-
Final response: what the agent returned and why it stopped
For example, a support agent’s trajectory might show that it retrieved a customer record, checked refund eligibility, requested approval, issued the refund, and notified the customer. The final message is only one part of that path.
Trajectory quality and task success are related, but they are not identical. An agent can follow the correct process and still fail because the underlying data is wrong. It can also produce the correct answer after following a flawed process. Trajectory evaluation focuses on whether the process itself was sound.
According to a 2026 survey of AI agent systems, evaluating an agent requires examining tool use, planning, and decisions across multiple steps, not just its final answer. This step-level view helps teams identify where a path broke, whether the agent chose the wrong tool, repeated unnecessary actions, or made an unsafe decision. Once the full trajectory is visible, the next question is which parts of that path should be measured.
AgentBench puts this approach into practice by evaluating agents across multi-step interactions rather than judging only a single response.
Ebook — Inside Atlan AI Labs & The 5x Accuracy Factor
Trajectory scoring is only as good as the context behind it. See what actually moves agent accuracy inside Atlan AI Labs.
Get the EbookWhat dimensions matter when evaluating an agent’s trajectory?
Permalink to “What dimensions matter when evaluating an agent’s trajectory?”Once you have captured the trajectory, evaluate it across four core dimensions, the same context layer evaluation criteria that determine whether an agent’s output can be trusted in production. Each one reveals a different type of failure.
| Dimension | What it measures | Example failure |
|---|---|---|
| Tool correctness | Did the agent select an allowed and appropriate tool, use valid arguments, and interpret the result correctly? | The agent calls a deprecated endpoint or sends the wrong customer ID |
| Step efficiency | Did the agent complete the task within a reasonable number of calls, retries, tokens, and time? | The agent makes 15 tool calls when the reference path requires four |
| Plan adherence | Did the agent follow a coherent approach and change course only when new information justified it? | The agent abandons a valid plan, tries unrelated tools, and later repeats its original steps |
| Safety of intermediate actions | Did every action follow approved rules for giving AI agents access to enterprise data, including approval requirements and limits on irreversible changes? | The agent writes to a production database without approval and later deletes the record |
NVIDIA’s guidance on agent evaluation recommends measuring tool-call accuracy and trajectory efficiency separately:
-
Tool-call accuracy – tool selection, argument validity, and schema compliance
-
Efficiency – calls, retries, tokens, and latency per successful task
Anthropic’s guidance on building effective agents also recommends starting with simple, composable patterns and adding complexity only when it produces a measurable improvement. This makes unnecessary steps and the use of tools easier to identify.
But even well-defined dimensions need a baseline. Without a trusted reference trajectory, teams still have to guess which tool was correct, how many steps were reasonable, and whether the agent followed the right business logic.
Where does trajectory grading break down?
Permalink to “Where does trajectory grading break down?”Trajectory grading breaks when teams can see what the agent did but lack a trusted standard for deciding whether those actions were correct. The reason for this failure is twofold.
The first failure mode is the absence of an approved reference trajectory. A trace can show every tool call, argument, result, and state change, including agent harness failures and anti-patterns. But it cannot tell you whether the agent:
-
Used the right source
-
Applied the correct join
-
Followed the required approval step
-
Completed the task within reasonable limits
These are the same guardrails that matter when you secure multi-agent systems in the enterprise, where one agent’s unreviewed action can cascade into another’s.
Without a reference, the scoring rubric may reflect the evaluator’s assumptions rather than the organization’s actual business logic. A domain expert may know the correct joins, filters, and decision rules, but that knowledge often has not been encoded in a form the evaluator can use.
A reference trajectory does not need to prescribe one perfect sequence. It can define:
-
Required steps and checks
-
Acceptable alternative paths
-
Allowed tools and arguments
-
Prohibited or high-risk actions
-
Expected outputs and stopping conditions
Tracing tools and evaluation frameworks provide the record needed for grading. They show what happened, but the reference determines whether what happened was acceptable.
The second failure mode is using an LLM as a judge without business grounding. Using an LLM as a judge can determine whether a tool call looks reasonable or whether its arguments match the expected schema. It cannot know whether the agent used the correct join for a company’s revenue policy unless that business logic is part of the grading context.
The same limitation applies to hallucination detection. These methods can flag unsupported or inconsistent behavior, but they still need approved context to distinguish a technically plausible path from a business-correct one.
The NIST AI Risk Management Framework emphasizes documented evaluation, transparency, and ongoing monitoring. Applied to trajectory grading, that means retaining both the agent’s path and the criteria used to judge it.
The problem, then, is not a lack of trajectory data. It is the absence of a grounded standard for evaluating that data. Building that standard starts with the dashboards, queries, production traces, and expert decisions that already encode how the business works.
How do you build a reference trajectory that is grounded in real business context?
Permalink to “How do you build a reference trajectory that is grounded in real business context?”1. Capture expert workflows. Atlan’s Context Engineering Studio reads BI dashboards, SQL queries, and production traces to create testable questions with expected answers and paths, the same data quality discipline that keeps an agent harness trustworthy in the first place.
2. Add business meaning. Context Agents use lineage, SQL patterns, and usage signals to identify the right sources, joins, filters, and metric definitions. Domain experts review exceptions and approve the context an AI agent needs.
3. Make the reference reusable. Context Repos package approved logic as versioned units, providing a consistent way to structure context for AI agents. The Context Lakehouse delivers it through MCP, A2A, SQL, or APIs so agents across frameworks can use the same standard.
4. Learn from each run. Human corrections flow back into the reference, while MCP distributes updated context to connected agents. Drift detection and context versioning keep earlier scores explainable.
According to Atlan’s internal survey, 87% of customers rate AI-generated context as equal to or better than human-written context, with roughly 90% acceptance of agent-written outputs. Once that reference is versioned and reusable, the next challenge is applying it consistently when a task moves across several agents.
Tool — Context Layer ROI Calculator
Building reference trajectories takes real engineering time. Model the payback in fewer redundant tool calls and less manual re-grading.
Calculate Your ROIHow do you evaluate trajectories across a multi-agent fleet?
Permalink to “How do you evaluate trajectories across a multi-agent fleet?”Single-agent trajectory evaluation is hard enough. Multi-agent systems compound the problem.
In a multi-agent setup, each agent has its own trajectory, and handoffs between agents are themselves trajectory steps that need grading. When Agent A hands off to Agent B, the handoff can succeed or fail independently of either agent’s internal trajectory.
Without shared reference context, each agent evaluates its own path in isolation. Agent A might have taken a correct path to the handoff point, but if it passed the wrong context to Agent B, the overall trajectory fails. You can’t catch this by grading each agent separately, which is why debugging multi-agent systems requires tracing the handoff itself, not just each agent’s internal steps.
The fix is the same reference context served to every agent in the fleet. When context management spans multiple agents, trajectory scoring becomes consistent across the system rather than a patchwork of per-agent rubrics.
This is also where trajectory drift compounds. As agents scale in production, an undetected trajectory failure in one agent propagates to every agent that inherits the same context or tool configuration.
How do you start measuring trajectory this week?
Permalink to “How do you start measuring trajectory this week?”You don’t need a full evaluation platform to begin. This is where harness engineering and trajectory evaluation meet. Start narrow and expand.
-
Pick one agent and one task type. Choose a task with known-good outputs where you can verify the answer independently.
-
Log the full trajectory. Capture reasoning, tool calls, arguments, and observations. If you are still deciding how to build an AI agent harness, make this logging part of the design. Otherwise, add observability instrumentation to the framework you already use.
-
Have a domain expert write the reference path. What tool calls should the agent have made, in what order, with what arguments? This is the ground truth.
-
Score the agent’s path against the reference. Use the four dimensions: tool correctness, step efficiency, plan adherence, safety. Grade each step, not just the final output.
-
Encode corrections back into your context layer. Every failure the expert identifies becomes a signal that improves the reference for the next run. Context quality testing makes this systematic.
-
Expand to more task types. Once the first task type has reliable trajectory scoring, add a second. Build the reference set incrementally. Test your agent harness to ensure the evaluation infrastructure scales.
A small, verified reference set beats a large, unverified one. The goal is not to score every trajectory on day one. It’s to establish the pattern: real business context produces real reference paths, and real reference paths make trajectory scoring trustworthy.
Tool — AI Agent Context Readiness Checklist
Ready to see where your own agents stand? Run your harness against the same readiness checklist enterprise teams use before scaling trajectory evals.
Get the ChecklistWhat’s the bottom line on trajectory evaluation?
Permalink to “What’s the bottom line on trajectory evaluation?”Most teams grade agents on whether they got the right answer, then wonder why the same agent falls apart on a slightly different question next week.
Trajectory evaluation is what tells you whether the agent understood the task or stumbled into the answer. That distinction only resolves against a reference: the actual sequence of tool calls, joins, and business logic a domain expert would have used.
Without codified business context, “the right trajectory” is a guess. With it, trajectory scoring becomes checkable, versioned, and something that improves every time an agent runs.
Tracing shows you the path. Context tells you if it was the right one.
Get started with Atlan’s Context Engineering Studio →
FAQs about how to measure agent trajectory
Permalink to “FAQs about how to measure agent trajectory”1. What is the difference between trajectory evaluation and output evaluation?
Permalink to “1. What is the difference between trajectory evaluation and output evaluation?”Output evaluation judges the final result. Trajectory evaluation examines how the agent reached it, including its decisions, tool calls, and intermediate results. This distinction reveals when an agent followed a reliable path and when it reached the right answer by chance.
2. What metrics measure a good agent trajectory?
Permalink to “2. What metrics measure a good agent trajectory?”A good trajectory is measured by tool correctness, step efficiency, plan adherence, and action safety. Tool-call accuracy measures tool selection and argument passing, while efficiency tracks calls, retries, tokens, and latency. Separate scores keep a strong result in one dimension from hiding a serious failure in another.
3. Can you use an LLM-as-judge to score agent trajectories?
Permalink to “3. Can you use an LLM-as-judge to score agent trajectories?”Yes, but only within limits. An LLM judge can assess tool relevance and argument plausibility, but it lacks the business context needed to define the correct path. Use it to supplement grading against a reference trajectory, not replace it.
4. What’s the difference between trajectory evaluation and tool-call evaluation?
Permalink to “4. What’s the difference between trajectory evaluation and tool-call evaluation?”Tool-call evaluation checks individual tool selections and arguments. Trajectory evaluation assesses how those calls, decisions, observations, and state changes work together across the full task. Individual calls can be correct even when their order or overall strategy is wrong.
5. How do you catch redundant or looping tool calls before production?
Permalink to “5. How do you catch redundant or looping tool calls before production?”Set tool-call, retry, token, and latency limits for each task, then compare test runs with an approved reference trajectory. Flag repeated calls with unchanged arguments, repeated failures without a new plan, and unnecessary backtracking. Apply the same checks before deployment and in production monitoring.
6. How do you keep trajectory evals up to date as your tools and business logic change?
Permalink to “6. How do you keep trajectory evals up to date as your tools and business logic change?”Version the business context behind each reference trajectory. When a metric, schema, tool, or policy changes, re-evaluate the affected trajectories and update the eval suite. Retaining prior versions keeps earlier scores explainable and auditable.
