Choosing Between Longer Agent Runs and Multiple Attempts

When an AI agent makes slow progress, should we let it continue or spend the remaining budget on a fresh attempt? For WTK, this is a question about allocating work within approved limits, not giving an agent more permissions.

The signal

When Agents Slow Down follows four general-purpose agent systems while they work on selected open-ended engineering tasks. The authors compare improvement per token with an independent-sampling reference. Their measured curves keep rising, but the late gains flatten relative to that reference across the reported systems and benchmarks.

In one Polyomino Packing task using Kimi K2.7 in Kimi Code, the authors compare ways to divide 100 million tokens. A three-session split, chosen from the task's measured curve, scored 264 joint Elo above one long session and 355 above ten short sessions. Elo here is a comparative rating of submitted solutions, not a percentage improvement or a cost saving.

What the evidence says

The broader study uses 14 selected tasks, deterministic task judges, and token accounting that includes cached tokens. Its independent-sampling reference relies on stated mathematical assumptions. These are selected open-ended engineering tasks, not evidence that every coding or business workflow benefits from the same allocation.

On the second allocation task, MoE Load Balancing, the one-, two-, and three-session allocations were statistically indistinguishable. The useful result is that allocation can be measured and may depend on the task, not that three attempts are generally best.

More attempts still need a reliable final choice

The paper measures the best scored solution found within a session and across grouped sessions. A deployment may not have an equally dependable way to identify its best result. Generating a better candidate is therefore separate from selecting it correctly, the problem discussed in Selection Is Not Verification.

Equal token budgets also do not establish equal bills or completion times. Tool execution, evaluation, result selection, and concurrent infrastructure can add costs. A practical comparison should report those costs and elapsed time alongside task quality.

What WTK could test

Consider a hypothetical coding task in an isolated repository, with permission to edit files and run tests but not deploy. One condition gives a single attempt the execution-token budget. Another divides that budget among a predeclared number of fresh attempts in separate copies of the same starting environment. These are independent attempts at the same task, not a collaborating agent team.

Before specifying this comparison, we would need to select a task and runtime and verify that they support isolated attempts, enforceable limits, and usable accounting. Our runtime-configuration note explains why a proposed control must match what the execution environment actually provides.

An approved experiment would keep the agent package, model, task inputs, tools, permissions, and evaluation criteria fixed. It would predeclare the allocation, repetitions, stopping rules, final-selection method, and separate evaluation overhead budget. Every attempt would remain in the record, including failures, interruptions, denied actions, and results that were not selected.

We would compare the quality of the result actually selected, not just the best candidate discovered afterward. Evaluation on tasks not used to choose the allocation would check whether a benefit extends beyond the development examples. A gain would not justify adoption if permissions widened, required records were missing, or failures were discarded. No clear difference would be a valid inconclusive result, not proof that the approaches are equivalent.

This is a proposed WTK comparison, not a completed experiment or an approved run. The September 2026 paper does not establish a WTK budget policy. Changes to the model, runtime, task, accounting, or selection method would require fresh measurement.

Source

Kaiyuan Liu and colleagues, When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis, September 2026. The WTK interpretation and proposed characterization are ours. No WTK experiment has been run.