When Should an Agent Stop Calling an Unhelpful Tool

An agent can describe a search result as useless and still call the same tool again. For an agent builder, the important question is whether that judgment changes what happens next, and whether another attempt has a reasonable chance of helping.

Judged Useless, Queried Anyway, an October 5 preprint by Chubin Zhang and colleagues, studies that gap. Its most useful contribution is a way to distinguish stopping in response to evidence from simply reaching a deadline.

Compare decisions at the same step

The authors use question-answering tasks with a small, question-specific knowledge base. Agents can search, look up passages or finish with an answer. The researchers simulate failures by substituting passages from other questions, including conditions where useful results eventually return.

They compare decisions after the same number of observations. In both histories, the latest result is judged useless. In one, every result so far was judged useless; in the other, an earlier result was useful. Matching the step count helps separate the evidence history from the amount of budget already spent.

The deadline test adds another check. Under persistent simulated failure, increasing the action budget from eight to sixteen moved the median answering action from eight to sixteen for three smaller models given a budget reminder. These medians cover runs that produced answers. Continuing longer did not deliver a meaningful success gain in that condition.

The result does not mean every extra query is wasteful. It shows why a run that eventually stops is not enough to establish that its stopping decision responds to evidence.

A fixed cutoff can prevent recovery

The authors also test a harness rule that tells the agent to finish after five consecutive useless judgments. It improves average task success in several comparisons, but its instruction is specifically to answer without further tool use, including from the model's own knowledge.

That matters when the source recovers. When useful observations return after five failures, the rule reduces success relative to the unaided agent by 21.3 percentage points for Qwen3-8B and 13.7 points for Qwen3-32B. The cutoff arrives before the agent can benefit from recovery.

These are controlled results from simulated knowledge bases, primarily using four open models. Some usefulness judgments are collected by replaying trajectories rather than appearing in the live agent's context. The measurements diagnose a relationship between judgments and actions; they do not reveal the model's internal reasoning or establish a universal five-call limit.

Changing course without inventing the answer

Consider an insurance-service agent looking for the policy wording that applies to a customer's coverage question. Several searches return irrelevant policy records. Repeating them indefinitely is unhelpful, but stopping retrieval and confirming coverage from memory would replace missing evidence with a guess.

A permitted next step could be refining the query, using an approved alternative source, requesting a missing policy number or leaving the question unresolved. Which response is appropriate depends on what is missing and whether recovery is possible. Stopping one search path is different from completing the customer's task.

This complements our note on plausible answers hiding failed searches. It also sharpens the budget-allocation question: more attempts are useful only if they improve the outcome under the task's requirements.

For WTK catalog packages and their target runtimes, the question is whether evaluation can distinguish evidence-responsive continuation from deadline-driven persistence, while also catching premature stops. Qualification would need to assess both errors.

The aim is higher-level oversight: people define acceptable evidence, delegated limits and exception conditions, while routine recovery happens within those limits. An unhelpful result should inform the next decision without becoming permission to skip required evidence.

Source

Chubin Zhang and colleagues, Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions, October 5, 2026, v1. See the matched-step analysis, budget-scaling comparison and late-recovery results, particularly Tables 6, 16 and 26. The insurance example and WTK question are our interpretation.