Learning Which Harness Changes Help Future Agent Tasks

An agent factory can retain a revised agent without learning much about how the revision helped. Keeping the proposed changes, their observed effects and the reasons for accepting or rejecting them could make the next improvement attempt better informed.

Harness², an October preprint from Northwestern University and Google Cloud AI Research, studies that second kind of learning. It improves the software around a frozen model, then retains experience to guide future harness changes.

Improve the task and learn from the improvement

The paper separates two loops. For the current task, an editor proposes changes to general working methods and domain-specific procedures. Candidates run on the same task under matched model and environment settings. A composer examines their changes and execution records, then merges, revises or rejects edits. Keeping the current harness is a valid choice.

Afterward, a separate reflection step records what those attempts revealed. The library retains executable components alongside lessons about the situations where they may help and the observations supporting them.

Each new task starts from the original execution harness, not automatically from the previous task's customized version. The improver selectively reuses earlier components and lessons. Model weights, stage instructions and the improvement algorithm stay fixed; the library evolves.

This is a concrete form of factory learning: improving the information used to propose and assess changes, rather than assuming the last selected agent configuration should become everyone's starting point.

Better evidence for editing is not a correctness guarantee

Harness² does not give its editor the benchmark's answer key, rubric or score. It compares what agents did and produced. The researchers use separate benchmark grading afterward to measure results.

Across three shuffled runs with Gemini 3.5 Flash under OpenCode, the full method scores 77.5 on WorkBuddy-Bench's three-domain average. Removing library use lowers that to 74.4. Those are benchmark score points, not percentages of complete tasks successfully delivered.

The paper also shows why the improvement record needs failures. In one coding case, an editor introduced an incorrect caching assumption. Its own checker reported success three times, while benchmark tests still exposed the error. Another selector chose an output scoring 55 despite having an alternative scoring 95.

These cases make the limitation practical: a proposed repair and its checker can share the same mistaken assumption. Repeated agreement within that process cannot substitute for checking the actual requirement.

The main comparisons use GLM-5 and Gemini 3.5 Flash under two adapted harnesses, with Claude Opus 4.8 editing. Additional probes and editor sessions consume compute, and model-based grading introduces its own uncertainty.

What this could mean for WTK

Consider a hypothetical procurement agent preparing a supplier-review packet from questionnaires, contracts and assessment records. A revision adds a source inventory and a script that checks whether every required section is populated. The new run produces a more complete packet.

The reusable lesson might concern finding omitted source files and checking output completeness. It should not carry a previous supplier's approval status into the next review, or treat populated sections as proof that their contents are accurate.

For WTK, the question is whether factory records can preserve that distinction: what changed, which behavior improved, what remains uncertain and when the lesson is relevant again. Catalog packages and compiled target runtimes still need evaluation of the complete workflow. A lesson about preparing approval evidence does not grant an agent permission to approve the supplier.

People can define goals, permitted changes and release requirements while automation proposes revisions and gathers evidence. The useful next question is whether accumulated experience improves those proposals without weakening independent evaluation as packages, models and runtimes change.

Source

Ruiyao Xu and colleagues, Harness²: Recursive Agent Harnessing for an Open World, arXiv:2610.14224v1, submitted October 9, 2026; PDF dated October 12. Sections 2-4 and Appendices B-F and H support this interpretation. The procurement example and WTK question are ours.