A Plausible Answer Can Hide a Failed Search

An agent can have the right tools, call several of them, and still finish with a plausible value taken from the wrong record. That is a grounding failure, even when the final action looks reasonable.

WTK has already covered what should happen when a required evidence tool is missing or the available evidence cannot make a claim answerable. This note addresses a different failure: the tool works and relevant evidence exists, but the agent stops searching after finding the wrong plausible record.

The signal

SPIEval evaluates mobile-assistant tasks in which required information is scattered across synthetic personal records. The agent must formulate searches, connect records, resolve ambiguity, and then invoke an execution tool with the supported parameters.

The useful mechanism signal for WTK is retrieve-or-escalate: do not treat a likely value as located evidence when the search path did not establish it.

The evidence

The benchmark contains 250 human-curated tasks, 4,335 records across ten apps, and 21 retrieval and execution tools. The best of nine evaluated models reached 57.3 percent accuracy and the weakest reached 16.4 percent. Removing retrieval and supplying all task-relevant records raised mean accuracy from 35.5 to 66.8 percent, while removing pagination alone moved it only from 35.5 to 36.0.

For the three strongest configurations, 79 percent of categorized failures contained an incorrect parameter value. The authors connect that pattern to premature commitment: agents often stopped on a plausible record rather than continuing to localize and verify the required one. Fewer than two percent of retrieval actions used the available regex or fuzzy-search modes.

The boundary

This is external benchmark evidence, not a WTK result. The records are synthetic, the tasks are mobile-assistant operations, and correctness is defined by author-created gold tool calls. The error analysis does not prove that one retrieval policy will repair WTK research or coding work. More search can also increase privacy exposure, cost, and irrelevant context.

The builder impact

WTK should distinguish three states that a fluent answer can blur: the required record was localized, the search ended without localizing it, and multiple records remain plausible. A receipt should bind the claim to the query sequence and disambiguating fields that selected one record. This is a search-termination decision, not another missing-tool control. If localization remains incomplete, the run should ask, narrow its scope, or return an unresolved outcome instead of converting confidence into provenance.

The WTK test

Compare the current WTK retrieval behavior with one candidate receipt-bound retrieve-or-escalate policy on a frozen synthetic corpus. Keep the goals, tools, permissions, model, evaluator, budget, and attempt order fixed. Seed exact matches, tempting near matches, conflicting records, absent records, and cases where continued searching would exceed authority.

Measure supported-answer accuracy, false supported answers, correct unresolved outcomes, unnecessary queries, scope violations, latency, cost, and catalog result. Count every attempt. The candidate succeeds only if it reduces false support without lowering correct completion or expanding retrieval authority. A tie or mixed result is inconclusive; any scope expansion or evidence laundering is a failure. This is a proposed WTK experiment, not approval to run it.

Still unknown

We do not know whether the current WTK path exhibits this failure often enough for the extra receipt and escalation logic to earn its cost.