Choosing Signals and Triggers for Agent Improvement

An automated improvement process needs a reason to start, not just a technique for making changes. For WTK, the important question is whether responding to a signal produces a better outcome than leaving the system unchanged or using a simpler response.

Imagine a deployed agent's task-failure rate rises. That could justify investigation, but it does not yet tell us whether to change instructions, repair a tool, retry the task or replace the model.

Separate the signal, trigger and response

We use three practical distinctions. A signal is an observed event or measurement. A trigger policy decides when that signal starts work. An adaptation technique determines what changes.

These are our organizing terms, not an industry taxonomy established by the papers below.

The same signal can justify different actions. Starting a review should not silently authorize deployment.

When a simple schedule beats a reactive trigger

Sawan Dasari's August 2026 preprint, When to Retrain, compares scheduled retraining, an error threshold and ADWIN, a statistical change detector. It includes a no-retrain baseline, limited retraining budgets and delays before replacements take effect.

For its static linear classifier, periodic retraining outperformed the tested reactive policies under abrupt and gradual drift in the synthetic benchmark. Noisy threshold crossings could consume budget before the important change arrived; conservative detection could leave budget unused. Reactive policies had better observed results under recurring drift, but that reversal did not survive the study's multiple-comparison correction.

With per-example learning enabled, full-retraining policies showed no practically significant differences across the reported comparisons. Crucially, “no retrain” still included those incremental updates.

These are controlled streaming-classification results, not LLM-agent results. Labels arrived immediately, retraining had a fixed unit cost, and real-data arms used constructed streams with only three configurations. The paper does not establish that schedules are generally superior. It shows why a detector's response needs evaluation alongside the learning system and resource constraints.

After failure, changing the next attempt can help or hurt

September's VRL-Bench, by Yu Bai and colleagues, asks whether failure-derived memory improves computer-control agents beyond memory-free retry. Its primary comparison gives three models six complete attempts on MiniWoB and WebShop, without changing model weights.

Existing memory updates improved observed success in some settings and reduced it in others. Dynamic Cheatsheet and ACE were adapted to within-task failures, so these results do not judge their original cross-task use.

The proposed VEX² scheduler uses failure records and remaining attempts to suggest which approach to try next. It had positive observed gains over retry in all six settings, but only four setting-specific confidence intervals excluded zero.

Trial limits were matched; tokens and model calls were not. The scheduler's allocation was advisory text, not a runtime-enforced plan. This supports studying response selection, not assuming reflection is always beneficial or cheaper.

What this adds to WTK

Our budget-allocation note asks how to divide available work. This adds an earlier question: what observation warrants starting or redirecting that work?

For WTK's catalog and compilation approach, we would want a proposed improvement to identify its signal source, trigger rule, affected agent or team package, permitted response and target environment. Telemetry means recorded operational measurements, not necessarily trustworthy evidence: missing records or agent-authored explanations could mislead the decision.

Before proposing an experiment, we need to identify an actual WTK trigger path and its current behavior. Questions include duplicate events, noisy thresholds, cooldowns, budget exhaustion and whether ordinary retry already solves the problem. These are design questions, not additional findings from either paper.

Our research method keeps evaluation and promotion separate. Neither paper demonstrates WTK improvement or current frontier-model performance; changing the model, runtime, feedback or budget would warrant fresh measurement.

The useful starting point is simple: define what starts the work, what may change, and how we will tell whether responding helped.

Sources

Sawan Dasari, When to Retrain, August 19, 2026, arXiv:2608.19488v1; Yu Bai and colleagues, VRL-Bench, September 11, 2026, arXiv:2609.12404v1. Both are preprints linked above. WTK has not reproduced these results.