Checking Whether an AI Agent Understands the User's Goal

An AI agent can follow its supplied goal and still deliver something the user does not want. Anthropic's Project Swap shows why we should check the agent's interpretation of preferences separately from how well it acts on that interpretation.

Consider a hypothetical travel helper authorized to shortlist flights. It might stay within budget but recommend a cheaper flight with an overnight connection when the traveler would pay more to avoid that stop. Here, the user is the principal: the person the agent represents. Checking goal understanding means comparing the agent's recorded priorities with that person's stated choices, not claiming access to their private intentions.

What Anthropic tested

Anthropic's Project Swap put 201 employees into a controlled book-trading market. Agents formed a ranking from a short intake conversation, while the participants' own rankings of up to ten books were kept out of the agents' trading inputs and used to score outcomes. The ranking comparison included 188 participants who submitted rankings.

The agent-built ranking agreed with the participant on 61% of ranked book pairs, compared with 50% for random guessing. That is useful signal, but not a guarantee that the ranking carries the decisions that matter most to the person.

Where the outcomes diverged

The scoring placed a participant's top-ranked book at 1 and last-ranked book at 0, with equal steps between ranks. The report's best feasible assignment scored 0.89 against participant rankings. The decentralized market scored 0.55. When the authors recomputed the best assignment from Claude's inferred rankings but scored it against participant rankings, it reached 0.60.

Under that design, the authors attribute 85% of the observed 0.34 shortfall to preference representation and 15% to bargaining. They also found that the largest Haiku-to-Opus difference on Claude's own ranking metric, 0.12, became 0.01 when scored on participant rankings. All these trading comparisons used the same inferred rankings. This is evidence about model choice during bargaining, not a test of whether a stronger model could elicit better preferences.

What this does not establish

This was a controlled Anthropic employee book swap, not a WTK study. The participants were not representative, rankings were unincentivized, and the scoring turns ordinal preferences into a linear utility scale. The 85% figure is a result of that setting and method, not a universal split between goal understanding and execution.

The report also does not say that systems should collect more personal data, infer hidden preferences, or expand their authority. An inferred goal record is not consent, and a better predictor is not permission to act.

What this adds to WTK

Our earlier note on acceptance criteria missing the goal asks whether the evaluation omits an important requirement. This note examines a related but earlier question: did the agent's input capture the user's priorities before it selected an action?

For WTK's catalog and compiler approach, evaluating an agent or team package in a particular runtime does not establish that every future user's objective will be represented accurately. A qualified package and a faithful task brief are different things.

For consequential delegated work, we think the evidence record should name the principal, decision scope, source of each goal field, and correction route. It should distinguish a policy-compliant execution from a separately measured check of goal fidelity. Our research method is useful here because it requires conclusions to stay inside the evidence that produced them.

A proposed WTK comparison

If a human selects this question for a non-production study, we would compare a frozen current WTK path with the same path plus a small, predeclared confirmation of representative choices. The agent or team package, model, runtime, tools, permissions, scoring rules, budget, and escalation path would stay fixed.

The comparison would use consented or synthetic fixtures, not hidden preferences or live customer decisions. It would retain corrections, abstentions, overrides, goal-field provenance, target receipts, and outcomes. Task completion and agreement with the declared objective would be measured separately. Before attempts, we would fix the fixtures, repetitions, stopping rules and minimum useful improvement, then retain every attempt. No improvement, excessive correction burden or weaker permission enforcement would count against adoption. A mismatch would narrow the claim. It would not create a new permission or alter an allow, deny, ask, or degrade decision.

Still unknown

We do not yet know which representative checks are informative enough, how much correction burden is acceptable, or whether this comparison improves outcomes in WTK. Those are questions for a separately approved experiment, not claims this note can make.

Our takeaway: evidence that an agent followed a supplied goal does not establish that the goal accurately represented the user. That distinction belongs in the evidence before a system speaks for someone else. See our open questions for the broader work of connecting goals, evaluation, and accountable deployment.

Source

Anthropic, Project Swap: What happens when agents trade for us?, September 24, 2026, 26-page report, especially pages 8-11 and footnotes 12-15, with a 15-page appendix. The reported results above are from the source. The WTK interpretation and proposed comparison are ours.