Checking How Agent Updates Use Reference Guidance

A task score can hide a change in how an updated model uses guidance. For agent builders, that matters when the same component handles ordinary work and difficult cases supported by a runbook.

Consider a hypothetical broadband provider updating a troubleshooting agent. Most connection faults follow a short diagnostic sequence. An unusual equipment fault needs an additional check described in an approved support note. Better results on routine faults would leave an important question unanswered: does the updated agent still use that note correctly when the unusual fault appears?

Keeping the case and note unchanged makes the comparison easier to interpret. Otherwise, a better answer might come from better guidance rather than a better model.

What ReSAIL measures

The September 30 preprint ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation, by Shengjie Jin and colleagues, studies model training across repeated cycles. A model learns from behavior supported by extra experience summaries, then operates without them. The updated model becomes the next teacher.

One diagnostic reuses the same summaries and initial states across 128 ALFWorld simulated household tasks. With Qwen3-4B after three cycles, training that preserves behavior without summaries scores 65.9% without guidance and 62.2% with it. ReSAIL's alternative, which preserves behavior conditioned on summaries, scores 67.2% and 70.3%, respectively.

The methods differ by 1.3 percentage points without guidance and 8.1 with it. Those are means over three decoding seeds, not independent training runs, and the paper gives no uncertainty interval for the differences. The diagnostic measures using fixed summaries, rather than generating summaries or improving subsequent teaching. Its hindsight-derived summaries also differ from an ordinary support runbook.

What this adds to WTK qualification

For WTK's catalog packages compiled into a target runtime, the useful idea is to distinguish the conditions under which a component must perform. If a package depends on supplied guidance, its assessment needs to cover that use alongside ordinary task completion.

Our model-update lineage note asks which experience and teacher produced a change. ReSAIL adds a way to examine the changed component while holding its reference material steady. Applying that comparison to a harness or package revision would be a separate engineering extension of the paper's model-training work.

In the broadband example, success means reaching the correct diagnosis and permitted next action. Mentioning the support note is insufficient. The note also cannot grant permission to change customer access settings; the deployment's authorization still applies.

Keep oversight focused on consequential change

People can define the required diagnostic outcome, approved guidance and action limits while routine troubleshooting proceeds automatically. A proposed update then needs evidence that it preserves those requirements in the situations where guidance matters.

Our success-definition approach keeps those requirements outside the improvement process's control. If the component will also help produce future learning material, that role needs its own evidence. Using an existing explanation well and producing a reliable new explanation are different capabilities.

The practical question for WTK is whether an update preserves the behavior each assigned role needs, including how it responds to the same supporting evidence.