Evaluating What an Agent Said, Did and Left Behind

An agent can tell a customer that a request is ready while important work remains unfinished. Even a completed request leaves another question: did the agent stay within the actions it was allowed to take?

Divya Santhanam's October 15 LangChain article, The Most Common Questions About Agent Evals, Answered, describes three useful things to inspect: the final response, the sequence of messages and tool calls, and the state of the environment after the run. For builders, the value is seeing why these can deserve different judgments.

One insurance claim, three questions

Imagine an insurance-claim intake agent asked to collect the incident details and required photos, then route the complete claim to an assessor. In this hypothetical workflow, it may request missing information but may not approve coverage or offer a settlement. Those are the example's assumed permissions, not an industry-wide rule.

First, read its reply. Does it accurately explain what was collected, what remains missing and where the claim goes next? A clear explanation is useful evidence about communication. By itself, it cannot show that the required photos actually reached the claim record.

Next, inspect its actions. Did it request missing information and route the claim without approving it or sending an unauthorized offer? This ordered record is often called the trajectory. A final claim marked pending assessment would not reveal whether the agent had already emailed the customer a settlement promise.

Finally, inspect what it left behind. Are the incident details and photos attached to the correct claim, and is the case in the assessor's queue? A checked box saying documents complete would not establish that the files are present and usable. The checks need to reach the actual requirement, as our note on matching checks to the required result explains.

These questions can disagree. The explanation might be clear and the claim complete, while the agent also sent an offer it had no authority to make. Combining those observations into one favorable score could conceal a disqualifying failure.

Choose the evidence before the scorer

Santhanam describes code checks for deterministic conditions, model judges for less easily specified judgments, and human review to check evaluators against the team's judgment. She also recommends isolated environments for tasks whose deliverable is a changed repository or system, so one evaluation run does not contaminate another.

Our interpretation is to decide what must be established before choosing how to score it. A model reading the reply cannot inspect claim records it never receives. A final-state check cannot reconstruct every action that preceded it. More evaluators help only when they have the evidence their questions require. Our trace-search note examines how relevant tool evidence can remain outside a review.

For WTK, qualification means evaluating a specified agent or team package in a specified environment. Could its review record keep communication quality, permitted actions and achieved state separately inspectable? Could it show unresolved evidence rather than letting a strong explanation compensate for a failed requirement?

People would define the required outcomes, permitted changes and consequences of failure, then review exceptions and evaluator disagreements. Higher-level oversight need not mean approving every tool call. These remain proposed WTK questions, not an implemented integration.

The source is engineering guidance, not a controlled demonstration that this evaluation design improves reliability. We have not replicated it. Under WTK's research method, changes to the task, tools, environment or evaluators would require reconsidering what the evidence establishes.