AI Agent Evaluators Need More Than Consistent Verdicts

A review gate can look reassuringly stable while failing to notice the difference it was meant to judge. For agent builders, a matching verdict for a cosmetic rewrite is useful only if the gate can still separate evidence that should produce different decisions.

The signal

In A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review, the authors compare reviewers of scientific manuscripts under two pressures. One pressure uses rhetorical rewrites intended to retain the underlying technical content. The other uses different papers, where a reviewer should not collapse into the same score.

Their RobustReview benchmark contains 60 anonymized submissions and 1,200 controlled variants. In the reported comparison, Gemini-3.5-Flash-Lite Standard had the lowest listed average absolute score change across rewrites, at 0.205, but its ability to distinguish papers was close to chance on the authors' discrimination measure, at 0.511. The authors call that pattern false robustness. A low change under rewrites can describe a reviewer that is stable, or one that is not responding to enough of the evidence to make useful distinctions.

The paper studies manuscript review, not WTK. Still, its contradiction is useful: exact-input replay and a clean evaluation receipt do not by themselves show that a qualification gate is adequate.

The builder impact

WTK already treats qualification as evaluation of a specified package in a specified target environment. A future claim about an evaluator should also show two properties separately. First, matched presentations that preserve the declared decision-relevant facts should keep the same outcome. Second, different fixtures that change those facts should remain distinguishable.

Consider an approval gate reviewing two deployment packets. Reformatting one packet or moving a citation should not change a decision when the authority, evidence, and target facts are unchanged. Replacing a required qualification receipt with an unrelated receipt should not receive the same verdict merely because both packets look polished. The first check measures consistency across equivalent presentations. The second checks whether the evaluator detects differences that matter. Receipt identity and required permissions should still be checked deterministically; a semantic judgment cannot make an unrelated receipt valid.

This is narrower than asking whether an evaluator repeats its decision for identical input, as A Clean Receipt Does Not Make a Stable Judge does. It also complements Defining Authority for Automated Agent Approvals: an evaluator can advise or check, but it cannot create the missing authority in a packet.

The boundary

The authors evaluate selected models and protocols on generated manuscript rewrites. The rewrites can still change content, structured extraction can miss details, and the reported fused approach is mainly tested in one model configuration. Their scores do not establish a WTK threshold, show that WTK fixtures have decision-preserving rewrites, or justify importing their prompts, weights, extraction format, or model ranking.

The WTK test

This note does not recommend an experiment or a new evaluator design. It names the evidence a later qualification claim would need. For one fixed decision family, that record would bind the package and target identities, the evaluator and rubric, a small fixture set, predeclared matched presentation variants, distinct controls, every verdict and invalid output, and the rule used to interpret disagreement.

The question is straightforward: can a fixed WTK gate preserve its declared decision across presentation-only variants while distinguishing fixtures that should differ? WTK has not run that characterization or adopted a threshold. WTK's research method keeps this external mechanism signal separate from a qualification result or a deployment decision.

Source and review boundary

A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review, arXiv:2609.39027v1, September 30, 2026. This is an external-research interpretation, not a WTK Finding, an approved experiment, or authorization to change a target.