The Tool Contract Is Not the Tool Call

An agent can have the right tool contract and still stumble over the way that contract reaches the model. A new benchmark finds that programmatic tool calls matched or beat native JSON calls for most tested models, which is useful precisely because it turns a supposedly interchangeable runtime detail into something builders need to measure.

The signal

In The Bitter Lesson of Tool Calling, researchers compared typed Python-stub tool calls with native JSON tool calls across 14 language models on BFCL v4. Programmatic calling matched or exceeded the JSON baseline for 11 of 14 models. Under parallel fan-out it did so for 13 of 14. In the paper's context-rot condition, where accumulated context makes reliable tool use harder, the JSON baseline fell 2.3% on average while programmatic calling remained stable.

The headline is not that one interface has won forever. It is that two ways of presenting what can be the same tool capability produced different results. The paper reports a 10.6% improvement for one evaluated model family, a useful number with a narrow home, not a general speed or safety claim.

The evidence

The benchmark exposes programmatic tools as typed Python stubs that a model invokes through code, with execution and results handled in a single agent turn. The comparison is about tool-call performance on BFCL v4 under the stated conditions, including fan-out and a context-rot test. It gives builders a concrete reason to treat the call representation as part of the harness, not just formatting around a stable capability.

WTK research views this as support for a distinction we need to keep sharp: a governed tool contract can be portable, while the runtime binding that carries it may not be behaviorally equivalent.

The boundary

This is an external benchmark, not a WTK experiment. It does not test whether permissions are enforced, whether a receipt proves the right action occurred, whether tool output is safe, or whether a package behaves the same on every target.

It also does not show that programmatic calls are best for every model, tool, context window, or production workflow. "Stable" refers to the paper's particular context-rot condition, and the reported gains belong to its benchmark and model set.

The builder impact

When comparing runtimes, hold the declared tool contract fixed and vary the call representation deliberately. Measure task success, required-tool completion, invalid actions, fan-out behavior, and the effect of accumulated context. Keep the harness, tool schemas, and scoring conditions visible so a better number does not hide a different task.

Our take: tool selection and tool qualification are separate jobs. A contract says what an agent is allowed or required to use. Qualification has to show how that contract behaves where the agent will actually run.

The WTK test

WTK should use paired fixtures that preserve the same governed tool requirement while changing only the target-local call representation. Each run should retain a sanitized receipt for the required action, score rule-by-rule conformance alongside task completion, and keep the result attached to that target rather than inheriting it from another harness.

That would test a bounded question: whether the representation changes behavior for this package, this tool, and these declared conditions. It would not turn a benchmark result into a maturity promotion.

Still unknown

We do not know which tool shapes or task types make the representation matter most, or whether the result survives safety-sensitive and state-changing actions. The next experiment is simple: vary one call representation at a time, preserve every other condition, and count both successes and invalid actions before choosing a runtime default.