Testing Whether AI Agent Checks Catch Important Errors

To find out whether a check catches important errors, deliberately introduce known faults and see which ones it misses. For WTK, that offers a concrete way to assess a checker before relying on its results: test faulty examples alongside known-good controls, then evaluate improvements on faults withheld during development.

The signal

In Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles, the authors tested KernelBench's official evaluation oracle rather than treating its verdict as the end of the story. They report that, on their witnessed pool of 7,384 GPU-kernel faults, the official five-input protocol detected 6,136, or 83.1%, and missed 1,248, or 16.9%.

The misses were not evenly distributed. The study reports that 8.7% of its arithmetic faults escaped the oracle, while 78.6% of its precision faults did. This is the uncomfortable bit: an aggregate score can look fairly strong while a specific fault family has a large blind spot.

The evidence

The researchers then selected compact input suites from a matrix showing which witnessed faults each input exposed. They split every problem's faults by identifier before selection and scored the chosen suite only on the other half. Their two-input suites detected 94.8% of held-out faults, compared with 82.9% for the official five random inputs on that held-out accounting.

More detection was not enough on its own. A reconstructed fuzzing baseline produced 107 false rejections of correct kernels. A test input that condemns a known-good implementation is not useful evidence that a checker became more discerning. It is a different defect, and the paper keeps that defect visible.

The boundary

This is a 2026 arXiv preprint about GPU kernels under the authors' stated H100, PyTorch, and CUDA conditions. It is not a result about WTK packages, qualification checks, target adapters, or deployment decisions. The measured rates depend on the paper's witnessed fault pool and mutation approach, and the appendix retains an unresolved reconciliation between 120 loaded rules and 124 counted rules.

The paper does not supply a universal coverage threshold or a ready-made mutation model for language-based agents. It gives us a mechanism signal: a checker needs separate evidence for what kinds of wrong behavior it can distinguish, and for whether its challenge cases also spare known-good behavior.

The builder impact

WTK already separates a package from its qualification evidence. We should apply the same discipline one level lower. A receipt that says a deterministic check passed should name the package, target, checker, inputs, and decision it covers. It should not quietly become evidence that the checker itself has enough fault coverage.

Our research method calls for bounded claims and retained attempts. The related question in testing agent acceptance criteria is whether a review asked the right questions. Here the question is narrower: can one declared checker detect a declared family of faulty subjects without rejecting valid controls?

Our earlier benchmark blind-spots note asks whether tests represent the intended requirements. This paper adds a measurement technique: seed defined faults and count what escapes, rather than infer coverage from ordinary passing examples.

The WTK test

We propose a small, optional checker-adequacy study for one load-bearing WTK deterministic gate. A human would first select the real package, target, checker version, and a deliberately small fault model. The study would freeze those identities, retain behaviorally distinct seeded faults, validate every challenge against a known-good subject, and choose any supplemental cases on one partition before scoring a held-out partition.

The measures would include missed seeded faults, false rejections of valid controls, exclusions, and the limits of the fault model. No result would qualify a package, alter a gate, promote an asset, or authorize deployment. It would only tell a reviewer whether the named checker has evidence of adequacy for the named fault model.

The takeaway is simple: a deterministic checker is worth having, but its own blind spots deserve measurement. A pass is evidence about the subject. Checker adequacy is a separate claim that needs its own test.

Source

The authors of Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles, arXiv v1, September 2, 2026. The WTK interpretation and proposed study are ours; WTK has not run the study.