A Stronger Critic Can Make Tool Use Less Reliable

A reviewer that looks smarter on paper can make an agent less reliable in the runtime where it blocks actions. That matters whenever we put a critic between an agent and a real tool: the critic is not a decorative safety layer. It is a component that changes which actions reach the target.

For a concrete example, imagine a support agent about to modify a customer order. A critic that catches a bad item ID is useful. A critic that repeatedly rejects correct order changes can leave the customer with fewer successful outcomes, even if every rejection sounds careful.

The signal

CAST evaluates a step-local critic before a tool-calling policy executes an action. The authors report both single-run success and the chance that all four repeated executions succeed across dynamic customer-service simulators. That second measure is the sharpest part of the result: a harness can look acceptable in one trajectory while failing too often to be dependable over repeated runs.

With the paper's fixed Qwen3-32B actor, no critic averaged 31.7% single-run success and 11.5% four-run success. A GPT-4.1 critic fell to 22.8% and 6.0%. A Qwen3-235B critic fell to 24.1% and 3.4%. The authors' learned 4B and 8B critics instead reached 31.8% and 15.9%, then 30.4% and 19.5%.

The paper offers one mechanism for the gap. In its action-label audit, the GPT-4.1 critic flagged 46.8% of judged-correct actions, compared with 13.6% and 11.4% for the learned 4B and 8B critics. A bigger reviewer was not automatically a better fit for the actor, tools, policies, and target context that made up this particular runtime.

The boundary

This is a preprint about the authors' Qwen models, learned critics, customer-service simulators, action labels, temperatures, and budgets. Their action audit uses the study's labels, not general human ground truth. The authors also show a critic approving an action with reversed item IDs, and note that their critic checks the current action rather than modeling downstream consequences.

So the result does not tell us that WTK has an unsafe critic, that a learned critic is required, or that either named model should review a WTK target. It does not grant a reviewer execution authority, prove operator authorization, or replace deterministic permission enforcement. It is a dated mechanism signal, not a WTK qualification result. WTK has not replicated it, and a changed model, tool set, target, decision policy, or budget would require a new test.

The builder impact

WTK research reads this as a target-qualification problem. A package can declare the permissions and a contract can preserve them, but the deployed reviewer still changes the compiled runtime's behavior. Its identity, policy, false denial rate, false approval rate, and repeated outcome therefore need evidence for that exact target.

WTK's evidence lifecycle and research and falsification method already point in the right direction: retain negative paths, bind evidence to the actual projection, and make promotion a governed decision. Our earlier note on why a portable package does not qualify its runtime made the broad target-specific case. CAST supplies a useful calibration test for the reviewer inside that harness.

The WTK test

Our proposed experiment holds one compiled WTK target fixed and compares a no-reviewer control with one identified advisory reviewer. It freezes package and contract digests, target, model binding, tools, permissions, goals, evaluator, seeds, and budget. It retains every proposed action, reviewer receipt, decision, revision, final target state, and repeated-run outcome.

Measure completion, policy violations, false denials, false approvals, cost, and repeated target outcomes against a predeclared target-local oracle. The reviewer fails the experiment if it merely changes behavior without improving the fixed target's repeated results, if it hides rejected actions, or if it creates its own authority. External research gives WTK a testable hypothesis, not a maturity promotion.

Source

Primary research: CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents, arXiv: 2608.30147. WTK reviewed the canonical 29-page arXiv v1 paper and treats it as external evidence for a proposed target-local calibration experiment.