A Matched Score Can Still Miss the Failure

A command-generating agent can look fine on a matched benchmark and still fail because of the path between its answer and the machine. That matters for builders because a package-level score cannot tell us whether the command was wrong, or whether the transport damaged it after generation.

The signal

QuoteBench: How Matched Scores Can Hide Command-Path Failures tests that seam with exact final-state validation. The authors replayed model replies through an added, deliberately unescaped parser on 56 one-shot Bash tasks from 14 incident-derived families. Across eight same-window configurations, the added parser lowered success by 55.4 to 73.2 percentage points.

That is the contradiction worth keeping. A matched execution score can conceal two opposing effects: transport damage and a model adapting when it is told about the boundary. For the reported GPT-5.6-sol result, a matched gap of minus 3.6 points hid minus 64.3 points of transport damage and plus 60.7 points of compensation after boundary disclosure.

The evidence

The paper holds the generated reply fixed for its replay comparison, then uses escaping at the interpolation point to reproduce the raw-path outcome. That makes the observed recovery after disclosure more interpretable: it must come from changed generation, not an accidental repair of the same reply downstream.

The deployment configuration also changed comparative results. The authors report one unambiguous model-ordering reversal among 26 comparable pairs, plus four comparisons decided by a single task margin. In other words, a command agent is not simply a model with a score attached. It is a model, a generation contract, an execution path, an operating point, and a final-state validator.

The boundary

This is a one-shot Bash benchmark with a deliberately unescaped parser. It is not a WTK package evaluation, evidence that WTK has a transport defect, or proof that every command-capable agent will behave this way. Disclosure helped in six configurations but had zero or slightly negative recovery in two, so it is not a general fix either.

The result also does not establish authority enforcement, safe tool use, or equivalent behavior across targets. Those are separate claims that need their own evidence.

The builder impact

Our take: do not let a successful command string stand in for a successful command path. If an agent emits a command through a wrapper, serializer, shell, remote runner, or tool adapter, the execution path is part of what the user receives.

WTK should therefore keep canonical package identity separate from target qualification. The package can declare the tool and its permissions, while the compiled target needs evidence that its actual transport preserved the intended behavior. A good comparison freezes the package, model, goal, authority, command contract, and final-state oracle, then varies only the declared transport.

The WTK test

For WTK, a bounded target-qualification fixture would retain the raw generated command, each transport transformation, the package and compiled-target digests, and the exact final-state outcome. It would include negative paths and report whether an observed failure began in generation or after the command left the model.

That is enough to make a careful claim about one declared target. It is not enough to call one model better in the abstract, or to promote a target because its package passed somewhere else.

Still unknown

We do not yet know which transport boundaries matter most across WTK target types, or how often a target can change an apparent model ranking. The next step is a predeclared, package-bound comparison across representative transports, with every outcome retained and no adaptation allowed to erase the damage it was meant to reveal.