Measuring AI Agent Reliability Across Repeated Business Workflows

An AI agent can complete a business workflow once and still fail to repeat that result. If the work changes bookings, payments or access records, selecting a successful retry can hide unreliable completion and the incorrect outcomes of failed attempts.

Consider a failure documented in Thinkingbox. An insurance customer asks for a payment extension. The agent retrieves a record showing two prior arrangements, the maximum allowed by the benchmark's policy, then grants a third. Every changing tool executes without an API error, and the agent reports success. The tools worked; the resulting arrangement violated the requirement.

What Thinkingbox measures

In Thinkingbox, Zhuochun Li and colleagues introduce a sandbox and 507 workflows across retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt starts from the same initial records in an isolated tool session. Agents can ask a simulated user for missing information and use tools to inspect or change the records.

The evaluator checks the required final database state and rejects wrong values, missing changes, and extra changes. It accepts different valid paths to the same required outcome. Thirty tasks also check essential properties of the final response.

The authors run 20 attempts per task. Kimi-K3's reported average attempt success is 57.37%, while 93.89% of tasks succeed at least once in those attempts. Yet only 68 of 507 tasks pass every attempt. Those measures answer different questions: can the agent find a successful path, and how consistently does it complete the work under the tested conditions?

A clean ending can conceal the wrong result

A separate retrospective analysis examines 79,853 failed trials in a common sample across 12 models. Of those failures, 67.24% end cleanly, include a changing tool call, and show no explicit error in the final tool response. That percentage concerns failed trials, not all work. It demonstrates the weakness of those particular completion signals when the required outcome is a persistent record change.

This complements Matching AI Agent Checks to the Required Result. That note asks whether a check reaches the requested conclusion. Thinkingbox adds a repetition question: even when the resulting records are checked, does one favorable attempt represent the rest?

What the evidence leaves open

The October 1 revision uses synthetic reconstructions of confidential business cases, not a representative sample of enterprise work. Each task has one accepted terminal state. Different execution paths are allowed, but workflows with several defensible final resolutions are excluded.

The simulated user is a cooperative GPT-5.4-mini deployment with a fixed objective and at most ten follow-up turns. Its phrasing varies across attempts. Most tasks, 477 of 507, do not score final-message accuracy, so correct records can pass despite an inaccurate explanation. The reported results concern that setup, not today's production failure rate or a universal model ranking.

The question this sharpens for WTK

WTK's catalog describes agent and team packages. A package defines the intended work and boundaries; the compiler prepares an execution form for a target runtime. Qualification evaluates a specified package in that environment. Our runtime-configuration note explains why that evidence belongs to the setup that actually runs.

Our take is a learning question: does a qualification record make clear whether it describes one discovered success, average completion, or repeated correct completion? Which required changes and forbidden effects does its evaluator inspect, and which communication requirements remain outside the check?

Answering those questions requires retaining unfavorable outcomes alongside successful ones, as our attempt-accounting log discusses. It would not make a catalog listing or compiled package a deployment approval. WTK has not replicated Thinkingbox, and this note recommends no experiment or implementation. Changes to the model, target, policies, simulator or outcome checks would require renewed assessment under WTK's research method.

Source

Zhuochun Li and colleagues, One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows, arXiv:2608.19741v4, revised October 1, 2026. Complete 56-page paper reviewed, especially Sections 3-5 and Appendices A, B, C.7, D.1-D.4. This note interprets external research; it does not report a WTK experiment.