Separating Reviewer Execution Errors from Invalid Results

A blocked agent review can have different causes: the reviewer may have failed to run, or it may have returned an answer that failed validation. WTK previously grouped those cases under the same execution-error label. A recent diagnostic change separates them while keeping both from counting as a passing review.

For example, a reviewer connection might throw an error before returning an observation. Another call might return a passing answer associated with a different evaluation. That answer cannot count toward readiness, even though the call completed.

The diagnostic change

The scoped change records whether the evaluator adapter, the software that calls the reviewer, returned before validation rejected its result. An exception before return remains an execution failure. A returned observation that fails the existing checks is classified as an invalid result.

The change does not relax those checks. Rejected evidence still produces an error observation, retains the expected evaluator and subject identity, and leaves the affected review non-ready. It adds no automatic repair or promotion path.

This is a concrete follow-up to A Gate Failure Has to Name Its Own Layer. That earlier record identified attribution problems across generation and scoring. This cycle addresses a narrower product boundary: execution of a reviewer versus acceptance of its returned observation.

What the checks establish

The regression check reproduced the ambiguous classification before the change and passed afterward. It exercises both an adapter exception and a returned observation naming the wrong evaluator, while checking that neither produces a ready result and that evidence references remain retained.

The candidate's recorded offline suite passed, with three skipped tests. A later integration record reports passing focused evaluation-plan checks after reconciliation of the merged change. These are implementation checks, not live reviewer trials or a measured improvement in recovery.

The independent review also retained an inconvenient result: one protected check failed on both the committed baseline and candidate because it used an unsupported schema field. A derived subset passed, but that subset is not a complete held-out pass. The dependency audit likewise reported existing advisories and was not clean.

What we no longer assume

An error verdict does not tell us whether a reviewer failed to execute, returned unusable evidence, or made a substantively poor judgment in an otherwise valid result. Keeping those questions separate avoids attributing every blocked review to the agent package under evaluation.

Our note on evaluator consistency and discrimination addresses the quality of judgment. This change concerns diagnostic attribution. A validly structured result can still be wrong, and a clearer error label does not make a reviewer accurate.

The remaining question

The practical benefit remains unmeasured: does the distinction help an operator or recovery workflow choose the right correction without unnecessary package edits or unrelated reruns? It could support higher-level human oversight by making exceptions easier to interpret, but this cycle does not demonstrate reduced operator effort, cost, or recovery time.

The lesson is limited and useful: preserve where review stopped while keeping the acceptance rule unchanged. WTK's research method keeps that completed engineering change separate from a future claim that recovery has improved.