More Test Cases or More Repeated Agent Runs?

When comparing two agent setups, should the evaluation cover more different cases or repeat the same cases more often? Both consume resources, but they reveal different sources of uncertainty in the comparison.

In Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads, Jingjie Ning, Xueqi Li and Yibo Kong compare two retrieval policies: one plans searches before seeing results, while the other adapts later searches to earlier results.

The authors divide nearly equal model-token budgets among distinct questions, repeated search attempts and repeated answers from fixed evidence. Broader question coverage produced a standard error about one-third lower than allocating more budget to repeated answers. Standard error measures uncertainty in the estimated difference between the policies, rather than the agent's accuracy.

In another control, lowering the model's sampling temperature made answers more consistent without a clear improvement in comparison precision. The results concern one model family and two factual-question benchmarks. Broader coverage also required more searches, so equal token totals did not establish equal bills.

Ask what each repetition tells you

Consider a hypothetical enterprise access-request assistant being updated to interpret an organization's account policies. Reviewers could repeatedly test a routine employee request or spend some of that budget on contractor access, expired approvals and requests with missing information.

Repeating the employee case helps show whether the assistant handles that case consistently. It does not reveal whether the update correctly handles an expired contractor approval. Adding that case improves coverage, but one attempt may still miss variation in how the assistant responds.

The outcome checks remain the same: apply the relevant policy, identify missing evidence and avoid granting access beyond the approved request. Required security cases should remain in the evaluation even if they occur rarely in ordinary work.

Our repeated-workflow note asks how consistently an agent completes a task. This paper adds a different question: how should those repetitions be distributed to make a comparison between setups informative?

Budget for the decision being made

For WTK's catalog packages compiled into target runtimes, an evaluation should identify which work the selected cases represent and which package versions are being compared. Its record should distinguish different cases from repeated attempts and repeated observations of the same evidence.

Our execution-budget note considers how to spend resources while solving a task. Evaluation spending serves another purpose: estimating whether a proposed change improves the required outcomes.

People can set the required coverage and decision criteria while automated evaluation gathers the evidence. Reviewers need the estimated difference, its uncertainty and the complete cost before deciding whether further evaluation is worth its expense.

Source: Jingjie Ning, Xueqi Li and Yibo Kong, Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads, arXiv:2610.05034v1, October 4, 2026. The access-request example and WTK interpretation are ours.