Checking Whether an Agent Repair Changes the Intended Behavior

A repair can improve an agent's average task score without showing that it fixed the problem it was meant to address. Before accepting the change, a builder needs evidence of what the agent now does differently.

The October preprint GUI-HARVEST makes that distinction part of an automated improvement loop. It changes the harness, the software around a fixed model that manages actions, context, recovery and completion, then checks both task scores and predicted behavior.

Describe the repair before judging it

Consider a hypothetical customer-service agent allowed to update a customer's mailing address, but not their payment details. It sometimes reports completion while the address form still contains unsaved changes.

Before evaluating a repair, the builder could state the expected behavior: when the agent tries to finish with an unsaved address, it should resolve the save and confirm the stored address before telling the customer the request is complete.

That prediction gives the review something specific to inspect. Did the unsaved condition occur? Did the agent save the change? Did the customer record contain the intended address afterward? If the condition never occurred, that run cannot establish whether the repair handled it.

This extends our distinction between what an agent said, did and left behind: the question now concerns a particular change and the behavior it was expected to produce.

What GUI-HARVEST checks

The researchers collect repeated executions of the same desktop task, including actions, tool outputs and screenshots. Before evaluation, the system records an observable prediction for each proposed repair. A validator then judges whether the execution evidence supports, contradicts or leaves that prediction unresolved, separately from the score.

Accepting a change requires code checks, no score decline on either the search or validation set, an improvement on at least one, and sufficient behavioral support. Each prediction needs at least one definite task verdict and more supporting than contradicting verdicts. Disagreement can therefore remain even when a change passes.

In the Qwen3-VL-32B-Instruct comparison, the complete method scored 53.18% on 201 held-out OSWorld tasks, versus 49.36% without the behavioral check. Both final evaluations used three executions per task and averaged raw evaluator scores, including partial credit. The paper does not report uncertainty for that difference or independently repeated optimization campaigns.

The behavior judgments also depend on model interpretation. The authors describe remaining errors where an action looks valid but affects the wrong object or state. In our address example, inspecting the stored customer record would add evidence beyond screenshots.

How this sharpens the WTK question

WTK catalogs agent and team packages and compiles them for supported execution environments. For a configurable workflow repair, we would want its review to retain the package version, expected behavior, triggering condition and evidence for the decision.

That complements testing harness changes on held-out tasks. Held-out evaluation asks whether improvement extends beyond development examples; the behavioral check asks whether the stated repair actually occurred.

People could define success, protected permissions and exceptions while automation gathers routine evidence. A repair that saves an address correctly still must not gain permission to alter payment details.

Our question is concrete: can WTK make repair decisions more informative by checking the promised behavior separately from the overall score, and returning unresolved or contradictory evidence for review?

Source

Geyi Yang and colleagues, GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution, October 1, 2026, version 1. The customer-service example and WTK interpretation are ours.