Trajectory evaluation

Trajectory evaluation is the practice of scoring the entire path an AI agent took to reach an answer, including every tool call, the arguments passed to it, each intermediate step, and every handoff, rather than grading only the final response. It is also called trajectory-aware evaluation or trajectory-based evals; the names describe the same shift in what gets measured. Where a conventional eval asks whether the reply was right, a trajectory eval asks how the agent got there.

The distinction matters because an agent that produces a correct reply is not necessarily an agent that behaved correctly. It may have called a write endpoint it had no business touching, retried the same lookup eleven times, or pulled a full customer record into an intermediate step that never needed it. Outcome-only grading is blind to all of that. Frameworks that support agent testing, including LangChain evaluation material published in April 2026 and DeepEval documentation from August 2026, now treat trajectory as a first-class evaluation target alongside final response, and TRAJECT-Bench exists specifically to benchmark trajectory-aware tool use.

What an agent trajectory contains

A trajectory is the ordered record of everything an agent did between receiving a request and emitting a response. It is not a transcript of the conversation. It is the execution trace underneath the conversation.

The core unit is the tool call: which function the agent invoked, the arguments it constructed, what the tool returned, and how long it took. A trajectory also captures the model's intermediate reasoning turns, the retrieval steps it ran, the branch it selected inside an agentic workflow, and any point where control passed to another agent or to a person. In a multi-agent system, the trajectory spans all participants, so a supervisor agent's delegation decision is part of the same record as the sub-agent work it triggered.

Why outcome-only evaluation misses real failures

Outcome grading rewards agents for arriving at the right destination and says nothing about the route. That stops being acceptable the moment an agent can act on external systems. Consider three agents that all return the same accurate order status. The first read the order from a read-only endpoint. The second read the order, then called a refund endpoint with a zero-dollar amount because a prompt fragment nudged it toward "resolving" the issue. The third looked up the wrong account, discovered the mismatch, and recovered. On a final-answer eval, all three score identically. Only the second is a live incident, and only the third reveals a retrieval problem.

The same blindness applies to efficiency. An agent that reaches the correct answer after fourteen tool calls costs several times what a three-call agent costs, which shows up as user-visible latency long before anyone notices it in an eval report. Trajectory evaluation makes the route a scored property, which is the only way these failures surface before customers find them.

How trajectories are scored

Trajectory scoring methods fall into a few families, and most serious eval suites use more than one because each family catches a different class of error.

Exact-match on an expected sequence. The evaluator defines the tool sequence a correct run should produce and checks whether the agent produced exactly that, in that order. This is the strictest form and the most informative when it passes, because it confirms the agent is deterministic on a case the team cares about.

In-order and any-order subset matching. A looser variant checks that a required set of tool calls appears in the trajectory, either preserving relative order or ignoring it entirely. Any-order matching is the right default when steps are genuinely independent, such as fetching an order and fetching a shipping policy before composing a reply.

Precision and recall over tool calls. Treating the expected calls as a reference set lets the evaluator compute recall, meaning how much of the required work the agent did, and precision, meaning how much of what it did was necessary. Low precision with high recall is a wasteful agent; high precision with low recall is one that gives up early.

LLM-judged trajectory review. Where many paths are acceptable, an LLM-as-a-judge pass reads the full trace against a rubric and scores whether the sequence was reasonable, each argument well formed, and no step unsafe or unnecessary. This is the only method that scales to open-ended tasks, and it inherits the usual judge caveats around rubric drift and position bias.

The multiple-valid-paths problem

Expected-trajectory matching is brittle, and it is worth being blunt about why. Most non-trivial support tasks have several correct routes. An agent can verify identity before or after retrieving the order, answer a shipping question from the knowledge base or the carrier API, ask a clarifying question or infer intent from context. A reference trajectory encodes one of these as canonical, and every equally good alternative gets marked wrong.

The practical consequence is a false-failure rate that erodes trust in the suite. Teams respond by loosening to any-order subset matching, by scoring only the steps that are genuinely mandatory such as an authentication call before a data read, or by reserving exact-match for a small set of high-stakes flows. A golden dataset of trajectories helps, but only if it is curated to include multiple accepted paths per case rather than a single blessed one.

Where trajectory evaluation fits alongside outcome and guardrail evals

Trajectory evaluation is a narrow specialization inside the broader discipline of AI evaluation, not a replacement for it. A complete suite still needs outcome scoring for whether the customer got what they needed, guardrail evaluation for whether the agent stayed inside policy, and hallucination detection for whether the claims it made were grounded. Trajectory scoring sits between them and answers the question the other three do not ask.

In practice it runs in the same harness as the rest of agent testing, on the same fixed case set, with trajectory metrics reported alongside outcome metrics so a regression in one is visible against the other. This is what makes it usable for iteration: a team can change a prompt or add a tool, rerun the suite, and see that final-answer accuracy held steady while average tool calls per resolution rose by four. That is a real regression, and no outcome metric would have shown it.