The Receipt Cannot Prove the Model Looked

A tool call can succeed, return a valid observation, and leave behind an execution receipt while the model answers as if it never looked. That makes evidence availability necessary, but not sufficient, for a grounding claim.

The signal

The Illusion of Visual Tool-Use asks whether observations returned by crop-and-zoom tools causally affect the final answer. The authors audit six models across five fine-grained visual perception benchmarks using interventions at three levels: policy, complete trajectory, and individual step.

They identify two failure modes. In Calling Without Looking, the returned observation has no causal effect on the answer. In Looking Without Planning, an observation is informative but the sequence of calls is incoherent. Their trajectory analysis finds that the aggregate accuracy gain from visual tool use is concentrated in a calibrated minority of rollouts.

That is a sharper problem than a missing tool call. The tool can visibly participate in the workflow while its evidence does not participate in the conclusion.

The evidence

The most useful part is the step-level intervention. Under a fixed trajectory prefix, the researchers replace one returned observation and measure whether the answer changes. Their Visual Evidence Gain measure is designed to isolate the contribution of that observation rather than crediting the tool policy for aggregate performance.

This matters because normal evaluation often sees only correlation. The call happened, the answer was correct, and the receipt was retained. Those facts establish availability and sequence. They do not establish that the answer depended on the observation.

The boundary

The study covers visual crop-and-zoom operations, selected multimodal models, and five perception benchmarks. It does not tell us how common the same failure is in text retrieval, code execution, database queries, or WTK packages. A changed answer under intervention shows dependence, but it does not prove that the original observation was correct, sufficient, or interpreted for the intended reason.

The paper therefore does not invalidate receipts. It identifies a claim receipts cannot carry on their own.

The builder impact

Our take: builders should separate three questions. Did the tool run? Was the returned evidence valid? Did the result actually depend on that evidence? Execution receipts help with the first. Deterministic source and schema checks help with the second. The third may require a predeclared causal probe on the cases where grounding is load-bearing.

Running that probe everywhere would be expensive and easy to turn into evaluation theater. The right scope is a small held-out set of claims whose correctness should change when their evidence changes.

The WTK test

WTK already binds tool observations and evaluator inputs to exact runs. A bounded extension would preserve an original held-out run, then execute declared variants that withhold, corrupt, or replace one grounding-critical observation while keeping the earlier trajectory fixed where the target permits. We would compare the cited claim, abstention behavior, evidence reference, and task verdict across every retained variant.

This would remain advisory evidence beside receipts and deterministic checks. It should not rewrite the original run, hide favorable or unfavorable variants, or become a secret retry path.

Still unknown

We do not yet know whether this intervention transfers cleanly beyond visual tools or how often a model can respond to a changed observation for the wrong reason. The next useful result is small and falsifiable: identify a grounding-critical fixture, change only its observation, and test whether WTK can distinguish evidence presence from evidence use.