Matching AI Agent Checks to the Required Result

An AI agent can perform a correct check and still leave the user's requested result wrong. The question is whether the check settles that result, or only something the result depends on.

Imagine an agent asked to report a company's profit margin. A reviewer recomputes revenue correctly, but never checks the cost figure or the margin formula. The calculation passed; the requested conclusion remains unsettled. This is a hypothetical example, not a WTK incident.

What VeriHarness observed

In VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks, Caiqi Zhang and colleagues give a verifier access to candidate artifacts, workspace files, evidence tools, and reusable checking instructions. The verifier uses the same underlying model as the agent that produced the work. It investigates disputed answers and separately challenges answers that all candidates share, then combines the records to select or revise a deliverable.

The mechanism gives verification something concrete to inspect. It also exposes a narrower failure than simply trusting agreement: an investigation can obtain valid evidence about the wrong point in the chain of calculations.

Appendix J examines APEX-Agents claims where all ten Claude Opus 4.8 attempts were graded wrong and the consensus challenger upheld their shared value. Table 7 attributes 70% of those surviving errors to checks of an input or intermediate quantity when the graded quantity lay downstream. Another 14% involved a different definition, period, or source version; 12% confirmed the wrong graded value; 4% involved omission or mapping noise.

Those percentages describe a selected group of errors. They do not say that 70% of tasks failed, or that 70% of all checks were misplaced. The paper does not report the raw number of surviving errors behind that table.

What the analysis cannot establish

The same model maps challenger records to rubric items. Benchmark grades identify incorrect answers; neither that mapping nor the grades are independent human ground truth. The taxonomy is therefore an attributed observation with classification uncertainty, not a universal failure rate.

The October 2026 preprint evaluates five workspace benchmarks using Gemini 3.5 Flash and Claude Opus 4.8, with ten candidate attempts per task and three verifier runs. The specific surviving-error analysis uses APEX-Agents and Opus. It does not test WTK packages, permission enforcement, or deployment approval. Model, grader, task, or checking-procedure changes require renewed assessment; this note makes no current model ranking or cost-saving claim.

Why the distinction matters for WTK

WTK connects an operator's goal to required evidence and evaluates a specified agent or team package in a specified execution environment. That evaluation is qualification. A record showing that a tool ran or a calculation matched its inputs cannot, by itself, establish every requirement attached to it.

Our note on evidence receipts asks whether an observation actually affected an answer. Here, the observation may be used correctly while supporting only an intermediate quantity. Our acceptance-criteria note distinguishes evaluator mistakes from missing requirements. VeriHarness adds a concrete case within evaluation: the final requirement can exist, yet the review stops before the evidence reaches it.

For builders, the useful review question is specific: what does this check establish about the requested result, and which dependencies remain unchecked? A revenue check can support the revenue input. It should not silently become evidence that the margin, period, or final report is correct.

What evidence would answer WTK's question?

We would need to inspect the requested claim, the quantity actually checked, the source and definition used, and the remaining steps connecting them. An unresolved final claim should stay visible even when an earlier check succeeds. Whether WTK's current qualification evidence distinguishes those cases remains an open question.

We are retaining that question, with no experiment recommendation or maturity contribution. WTK's research method keeps this external observation separate from a WTK result and from authority to change a gate.

Source

Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, and Chen-Yu Lee, VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks, arXiv:2610.00972v1, submitted October 1, 2026; PDF dated October 2. Complete 28-page primary source reviewed, especially Section 6 and Appendix J. This is an attributed Research Note, not a WTK Finding or an approved experiment.