Cheap Agent Decisions Can Lead to More Expensive Runs

A cheap model call can make an agent's completed work more expensive. The important cost is what the recommendation causes the rest of the workflow to do.

Naman Vats's October 17 preprint, Good Judgment, Bad Advice, tests that distinction. It adds small decisions about continuing, recovering and checking work to one terminal agent, then measures the whole run rather than the price of the decision alone.

A useful signal can produce an unhelpful intervention

The study keeps a Claude Sonnet 5.5 agent loop fixed and compares five versions: no added controller, deterministic rules, Claude Haiku 5.5, the typed decision model Jev 1.13, and a randomized controller.

Each controller is consulted at defined events, such as repeated command failures or a request to finish. Its choices can add a fixed investigation, recovery or verification message. Shared software guards retain the budget limits and bound repeated verification.

The randomized version samples choices using frequencies collected from Jev during development. This placebo helps ask whether informed recommendations add value beyond occasional nudges at the same types of decision point. It does not hold the eventual number of interventions identical.

Across 21 held-out Terminal-Bench 2.1 tasks, with three attempts per version, Jev and the no-controller agent both passed 73.0% of attempts. Jev's cost per attempt was 19% higher by the paper's paired geometric-mean comparison, with a 95% interval of 2% to 36% higher. These costs include model calls, not sandbox compute.

The pass-rate comparison remains uncertain: its interval spans roughly 11 percentage points worse to 13 points better. Equal observed rates do not establish equivalent capability.

The extra work cost more than the decisions

Offline, Jev ranked likely successful finish requests better than the rule and small-model alternatives. But its completion probabilities were too low for this task population, causing it to recommend checking work that was often already correct.

The online Jev calls accounted for just 0.11% of model spending. Longer agent runs accounted for the meaningful cost. Verification contributed, but the paper cannot causally attribute all the increase to verification alone. No controller chose the available abandon action, so that potential saving never materialized.

The scope matters: one agent loop, one decision-model snapshot, 21 tasks weighted toward easier work, and an unpinned mixture of serving hosts. The planned open-weight replication was not run. This is evidence about one controller policy, not a verdict on small models, source screening or tool selection.

Choose the intervention as carefully as the model

Consider a hypothetical procurement agent assembling a supplier-review packet. A small model recommends another document search because it is uncertain that the packet is complete. That recommendation might uncover a missing required certificate, or repeatedly rediscover material already present.

The useful question is not simply whether uncertainty was detected. Did the next search resolve a missing requirement, reduce a consequential error or merely add cost? Required evidence and permission checks should remain mandatory regardless of which optional advisory component is cheaper.

For WTK's bounded semantic decision roles, this makes the judgment-to-action policy part of the evaluation subject. A model's ranking quality, the rule deciding when to intervene, and the resulting agent work deserve separate inspection.

Qualification means evaluating the selected package in its intended runtime. Before assessing an added controller, we would need the actual supported decision points, permitted interventions and current baseline. Our runtime-configuration approach keeps that question tied to what can actually be deployed.

People can set outcomes, required checks, budgets and exception conditions while software handles routine decisions. A cheaper recommendation is useful when it improves that complete process, not merely when it is inexpensive to ask.

Source

Naman Vats, Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success, arXiv:2610.24649v1, October 17, 2026. Sections 3–6, Tables 1–3 and Appendices A–D support this interpretation. The procurement example and WTK question are ours.