Evaluating AI Agents Without Leaking Prior Task Outcomes

An agent may solve a task more reliably after receiving notes from earlier attempts. For WTK, the evaluation needs to distinguish that useful assistance from an improvement that works on tasks without prior task-specific help.

Imagine testing a coding agent on a bug it has not seen during training, while giving it a summary of earlier attempts at that same bug. The training separation is real. But the execution is not independent of previous work on the test task.

What the paper tested

In The Tasteful Agent, Wenbo Pan and colleagues construct 502 decision questions from engineering and research runs. A trained student scores 47.9% against its base model's 30.0% on engineering questions from tasks excluded from its training fold. A question counts as correct only when answered correctly in both option orders.

A separate execution test uses 41 SWE-bench Pro tasks across 11 repositories. Its advice comes from 98 decision points mined from earlier runs on those same tasks. The student selects recommendations without being trained on those tasks; its choices are not guaranteed correct. With student advice, reported executor success rises from 14.6% to 33.7%.

The paper distinguishes student-selected advice from a separate correct-advice condition. It also requires fresh checks of the resulting patch. The executor is Qwen3.6-27B through SWE-agent 1.1.0. These are preprint results under that configuration, not WTK results.

Two useful results, two different claims

Our interpretation is not that the authors concealed their setup. They describe the same-task advice explicitly. The distinction matters when deciding what evidence WTK would accept for a particular claim.

The student comparison asks whether learned judgment transfers to questions from tasks absent from training. The execution comparison asks whether an agent benefits from advice built around prior attempts at the evaluated task. The latter does not, by itself, establish performance without that task-specific assistance.

Fresh verification can establish whether the new patch passes the specified checks. It cannot establish that the agent solved the task without prior help. Neither claim replaces the other.

What this adds to WTK's existing approach

Our earlier note on testing harness changes on held-out tasks separates tasks used for development, selection, and final evaluation. This note asks a narrower question: after those tasks are separated, what information enters the agent's context when it runs?

A frozen agent can still receive task-specific hints through memory, retrieved notes, summaries, or another agent. Reviewing only its training data or package changes would miss that route. Here, “held out” means excluded from the information used to prepare and assist the evaluated run, not merely absent from a training fold.

For WTK, we would want an evaluation record to identify where advice came from, which task it concerns, when it was created, and who could read it. That is a proposed evidence requirement, not a claim that WTK currently detects every such exposure. Our research method provides the broader requirement to retain attempts and limit conclusions to their evidence.

A proposed WTK comparison

We propose comparing three conditions for one human-selected WTK task family:

• The current baseline without prior results from the evaluated task.
• The same setup with general advice developed only from separate tasks.
• The same setup with declared advice from earlier attempts at the evaluated task.

The third condition would measure assisted retrying. It would not count toward a claim about performance without prior task-specific help.

We would keep the agent version, model, runtime, tools, permissions, scoring rules, and budgets fixed. Before running, we would define task partitions, advice provenance, repetitions, stopping rules, and the minimum improvement worth considering. Every attempt, failure, interruption, and cost would remain in the record.

If general advice does not improve the baseline, we should report that result. If prior task information reaches a supposedly unassisted condition, that comparison cannot support the intended claim. Incomplete evidence may leave the result inconclusive.

WTK has not run this study. Selecting a protocol and authorizing execution remain separate human decisions.

The practical takeaway: use previous experience to help agents, but record when that experience concerns the very task being used to evaluate them.

Source

Wenbo Pan and colleagues, The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks, arXiv:2609.25804v1, September 22, 2026, Sections 5.2–5.3 and Appendix H. The evaluation distinction and proposed WTK comparison above are our interpretation.