More Thinking Just Bought a More Expensive Guess

Two of the cheapest ways to make an agent better are to hand it a declared list of its tools and to give it more room to think. A benchmark out this week says that on one axis, adapting when the structure of the world changes, neither move raises the floor.

The catch: agents search when they should deduce

ScrambleToolBench (DeCLaRe Lab, NTU Singapore, and A*STAR) is an interactive terminal benchmark that strips semantic tool schemas, so an agent cannot ride a helpful-sounding tool name and has to discover what a tool does by using it. Then it perturbs the world: mapping drift, stochastic action failures, temporal execution windows.

Succeeding at discovery does not mean you can adapt. Under structural change agents showed belief inertia, or fell back on exhaustive search instead of deductive strategies such as cycle tracing. Two results land squarely on us: increasing test-time reasoning amplifies brute-force search rather than enabling deductive recovery, and persistent memory reduces compounding errors but does not enable structural inference.

How WTK research views this

Both of our reflexes here are scaffolding reflexes. WTK gives an agent a tool registry, a declared list of the capabilities it may use, and when something fails our repair loop mostly re-runs with a stronger profile or more deliberation. On the structural-adaptation axis, neither raises the floor. That undercuts "escalate" as a repair move.

The nastier implication is about the registry itself. If part of an agent's competence is borrowed from a plausible tool name, then an entry whose declared behavior has drifted from its actual behavior is invisible. The agent mis-binds confidently, and the failure surfaces as a wrong answer rather than as a binding error. That is our top named blocker, whether a tool really does what its entry says it does, picking up a new way to go wrong. Not the tool-less case we already closed: here the tool is present and mis-described.

How WTK research proposes to solve it

Not by adopting the benchmark. WTK's posture is declared capability, so stripping tool semantics on purpose is the opposite of our design. We are taking the failure mode and one diagnostic question, aimed at a piece of the tool registry that already exists: before a capability may be called working, it has to pass a live smoke test. Does that smoke verify behavior, or only that the tool answers the phone? If it is only the latter, drift goes undetected, and this paper is the argument for a behavioral assertion in the smoke.

Our take: log and hold, not a new project. We act when our own dogfooding surfaces a mis-binding on a correctly registered tool.

Also on the radar

• Ask where the fault lives before you repair it. Model or Harness? (Scale AI) argues that the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair, and outcome-level labels cannot tell those apart. Its taxonomy makes the edge between two components the unit of analysis, with 41 failure modes, each carrying a fault side saying where the repair belongs. Judged by independent reasoning agents across four frontier models, the strongest reaches a Cohen's kappa of 0.76 against human category labels. Fault side is the routing key our self-repair vocabulary lacks, since today we mostly re-run harder without deciding which layer is broken. One guardrail: their point that grader failures can invalidate an evaluation must never become a lever for explaining away a red gate.

• Same model, different harness, very different score. LongHorizon-Harness (DreamX Team, Alibaba Group) keeps task state outside the growing context and updates it only with facts independently verified from the environment, checked by a read-only auditor before the next round. Same model, swapped harness: Qwen 3.7-Plus goes 51.8% to 80.7% on WeaveBench, 69.7% to 77.2% on Terminal-Bench 2.1, and 2.8% to 8.3% on OSWorld 2.0, while Claude Opus 4.7 goes 20.0% to 34.3% on an OSWorld 2.0 subset. We keep asserting that structure beats raw model without owning the model-fixed, harness-varied number, and this is that number. Two caveats we will not drop: the gains are task success only, so this says nothing about honesty or safety, and the results are self-reported on benchmarks the authors picked. The mechanism is already ours in spirit: advancing state only on verified environment facts is the grounding floor saying do not trust the agent's own account of what it did. No build.

The takeaway

If your agent looks sharp, check how much of that is the tool's name doing the work, then move something and watch. The uncomfortable read this week is that the usual dials, more memory and more thinking time, buy a more thorough search rather than a better inference. We would rather spend that effort making sure a declared capability behaves the way it is declared.

See you at the next catch.