Testing Agent Harness Changes on Held-Out Tasks

An agent can get better at the work it has already seen and still become worse at the job it was supposed to generalize to. A harness change therefore needs evaluation beyond the tasks used to develop it.

Imagine a support agent whose harness is updated after it repeatedly sends empty fields to a strict tool. The repair may be excellent. But if the same failures chose the repair and grade it, that score alone cannot establish whether the change helps with other requests.

The signal

StarHarness studies changes to the software around a fixed model: prompts, tool interfaces, skills, subagent structure, context handling, verification, and finish logic. It partitions reproducible tasks before search into visible diagnostic tasks, proposer-hidden selection tasks, and a final held-out set.

A candidate patch must pass scope, import, and smoke checks. It is accepted only by improving the hidden selection score, with an available verifier metric used only as a tie-breaker. Rejected, invalid, and crashed candidates are discarded and the previous accepted harness is restored. In the source's three benchmark settings, the fixed-model harnesses showed held-out gains of 31.7, 15.1, and 29.3 percentage points. Those numbers are a dated observation of that setup, not a WTK result.

The evidence

The paper reports that its frozen evolved harnesses also improved the listed GPT and Qwen configurations without being re-evolved. For WTK, the useful design question is how to separate these decisions: visible traces help diagnose a failure, a selection set hidden from the system proposing changes decides whether to accept a patch, and the holdout tests the final claim once.

In WTK, a target projection is a package adapted for a particular execution environment. Its evaluation results should not be assumed to apply to other environments. Target compilation and qualification evidence give us places to preserve that distinction. A changed harness should bind its package, contract, target, tools, permissions, evaluator, and task partition to the evidence that judged it.

The boundary

This 10-page preprint evaluates one evolving Stirrup harness on selected enterprise benchmarks, models, budgets, and evaluators. Its system comparisons are descriptive because prompts, tools, policies, and harness architectures differ. The authors also say the paired records cannot isolate which patch or tool caused a reported gain.

The study does not show that WTK should evolve targets autonomously, that a hidden set prevents all leakage, or that its performance gains transfer to a WTK package. It does not turn external services into a WTK requirement. WTK has not replicated the result, and a changed target, model binding, goal family, tool surface, or evaluator should trigger a fresh test.

The builder impact

Our take: passing a previously failed task helps diagnose and test a repair, but does not by itself justify release. Use separate evaluation tasks to check whether the change helps beyond the development examples. The release record needs a retained chain from the candidate diff through its selection decision, reversion history, compiled artifact, and final target evidence.

The WTK test

We propose one bounded comparison for an existing WTK target: the current compiled harness and one declared harness intervention. Before any change, partition reproducible goals by baseline failure mode, score, and verifier status into visible diagnostics, sealed selection, and one-use holdout sets. Freeze the package and contract digests, target, model binding, authority, tools, permissions, budget, evaluator, and stop conditions.

Measure completion, deterministic gate outcomes, invalid calls, false denials, cost, and evidence fidelity for every attempt. Retain every patch, scope check, smoke result, selection decision, revert, and target receipt. The intervention fails if it does not improve the sealed selection measure, widens authority, or regresses the held-out target without a declared reason. This is a proposed WTK experiment, not evidence that WTK has the problem or permission to run it.

Source

Primary research: StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments, Esakkivel Esakkiraja and colleagues, August 2026. WTK reviewed the complete 10-page primary PDF. This Research Note translates external evidence into a testable WTK question; it reports no WTK experimental result.