Making Evaluation Evidence Harder to Game

Agent systems produce a great deal of material that looks like evidence. Development runs, regression checks, evaluator judgments, operational traces, and successful demonstrations can all appear equally persuasive when they are presented together.

They are not equally authoritative.

This factory cycle focused on making that distinction explicit. The goal was not to generate more evaluation output. It was to prevent evidence collected for one purpose from quietly being used to support a stronger claim.

Current state

WTK is being built around the idea that goals define the work and evidence earns trust. That becomes fragile when every passing result is treated as interchangeable.

A development run is useful for finding defects and improving an agent. A regression check can show that a known failure has not returned. A held-out evaluation can test behavior the builder has not optimized against. A production observation can reveal how the system behaves under real operating conditions.

Each is valuable, but each answers a different question.

As WTK added reusable evaluators and richer observability, we needed to make sure those capabilities could not collapse these distinctions. A detailed trace should not become proof of correctness. A reusable evaluator should not assess a vague idea of quality. A successful development run should not become qualification evidence merely because it is available.

Changes

WTK now treats evidence according to its declared purpose and authority.

Development evidence supports debugging and improvement. Regression evidence checks known behavior. Held-out evidence may support qualification only under its declared conditions. Production evidence supports monitoring, incident review, and later improvement. Evidence does not gain authority simply because it is detailed or favorable.

Evaluators are also tied to the explicit obligations derived from the goal. This keeps an evaluator from issuing a broad approval while overlooking the specific outcome, evidence requirement, or operating constraint the agent was created to satisfy.

Attempts retain a terminal result. Passes, failures, inconclusive outcomes, and execution errors remain part of the record. A failed attempt cannot disappear because a later retry succeeded, and a semantic judgment cannot erase an objective failure.

Observability remains important, but its role is explanatory. It can show what happened, when it happened, and which components participated. It cannot establish that the answer was correct, the goal was accomplished, or the package deserves qualification.

Failures observed

This work exposed several ways an evaluation system can accidentally claim too much:

• an evaluator can approve general quality without testing the exact goal obligations;
• development evidence can be reused after the system has already adapted to it;
• interrupted or failed attempts can disappear from a favorable summary;
• evidence from an earlier package or execution condition can be associated with a later candidate;
• a rich operational trace can be mistaken for proof that the resulting artifact is trustworthy.

These are not exotic attacks. They are ordinary lifecycle mistakes that become dangerous when autonomous systems can generate large volumes of convincing output.

Assumptions removed

We no longer assume that a green evaluation means an agent is qualified.

We no longer assume that a reusable evaluator remains meaningful without being bound to the goal it is evaluating.

We no longer assume that more telemetry automatically creates stronger evidence.

We no longer assume that a later success makes an earlier failure irrelevant.

Evidence boundary

The mechanisms described here exist in current WTK development work and have been exercised through bounded validation. That supports a construction claim: WTK can represent different evidence purposes, retain terminal outcomes, bind evaluators to goal obligations, and limit the scope of qualification.

It does not prove that every evaluator is valid, that evidence cannot be manipulated, that every harness will preserve these boundaries, or that the additional rigor creates enough value to justify its cost.

Next hypothesis

The next test is to hold a governed goal and package constant while changing the model, harness, tools, or team shape. Evidence authority should remain explicit, failed attempts should remain visible, and qualification should stay bound to the execution conditions that produced it.

If those properties disappear when the execution environment changes, then the boundary is documentation rather than governance.