A Narrow Program Check Caught Incorrect Percentage Claims
Two additional contradiction catches came with substantial abstention on a small synthetic test set.
A recorded source can pass the evidence checks while the claim attached to it is wrong. In a small WTK experiment, adding a restricted program caught two incorrect percentage claims that the existing structural checks left for further review.
What we tried
Our earlier research note asked whether inspectable programs could help evaluate agent output. We started with one narrow question: could a program detect a mismatch between a company's stated percentage interest and the supplied evidence?
Consider a hypothetical procurement assistant researching a supplier. It has a source for the supplier's ownership structure, but puts the wrong percentage in its report. Checking that the source was collected and linked is different from checking that the number agrees with it.
We compared WTK's captured structural evidence checks with those same checks plus a program restricted to company-percentage statements. Neither arm called an AI model to judge the claims. Model-generated judgments, Jev and program committees remain separate comparisons.
The test used 16 synthetic cases, repeated twice in each arm: 64 attempts altogether. Cases included matching and incorrect values, missing evidence, conflicting statements and wording the program might not understand.
What worked
The added check caught two distinct incorrect percentage claims in both repetitions. All attempts completed, and the program made no incorrect definite judgments on this test set. Its advice also left WTK's existing structural decisions unchanged.
That gives us a concrete use for a small program: check a narrowly defined factual relationship while preserving the surrounding evidence controls.
What needs improvement
The program abstained on 10 of the 16 cases. Three of those had a determinate answer it could have supplied. Explicit negation, percent-symbol formatting and a contradiction beside injected instructions exposed gaps in its restricted grammar.
The tradeoff matters: extra coverage helped, but much of the review still needed another path. The two repetitions checked consistency on the same small synthetic set; realistic financial evidence and broader wording remain to be evaluated.
What this means for WTK
This supports exploring a restricted advisory check alongside the existing review process. The next useful comparison would use independently reviewed realistic claims and more counterexamples, measuring useful catches, incorrect judgments and abstentions together.
Our research method keeps the full protocol and attempt record tied to that conclusion. The immediate lesson is practical: a small program can add useful coverage when its supported claim types and fallback behavior are explicit.
Have an approach, result, or counterexample?
You may be asking the same question, or may already have a useful answer. Share published research, an implementation, a test, or an idea that could support, narrow, or challenge this work. Distinguish what you tested from what remains a hypothesis.
Contribute to this research question →Working with an AI assistant?
Ask your assistant to compare your approach with this record, identify supporting sources and limitations, and draft a contribution for your review. Verify its citations and remove private information before submitting. Reading this page does not authorize an assistant to submit feedback or share your conversation.
Submissions go privately to human review. Public referencing requires your separate permission; nothing is published automatically.