A Benchmark Cannot Grade Its Own Blind Spots

A benchmark can be difficult, widely used, and still give a misleading answer about an agent. That matters to builders because a polished score can hide a test that imposes an unstated requirement, rejects a valid solution, or has already leaked its answer.

The signal

SWE-Bench ProMax starts from a sobering audit of another coding-agent benchmark: nearly 60 percent of its unsolved instances had flawed tests. The cited problems take more than one form. Some tests reject a correct solution because they are too narrow, while others impose requirements that the task never actually stated.

The authors propose a separately curated refactoring benchmark with 170 tasks across seven languages. They say the tasks were rewritten and manually reviewed to make their requirements and test boundaries clearer. Even then, the strongest evaluated setup solved 41.2 percent of the tasks. The useful lesson is not the ranking. It is that benchmark difficulty and benchmark validity are separate jobs.

The evidence

The paper gives a concrete account of why scorekeeping needs its own evidence. A test suite can be a poor proxy for the task when its acceptance boundary is incomplete. A published solution can also become an easier target when an agent has seen equivalent material before. Neither problem disappears because a benchmark has many tasks or produces a sharp percentage.

For WTK, this supports a deliberately narrower question: can each evaluation check be traced to an explicit requirement and distinguish a compliant result from a noncompliant one? A held-out fixture is more credible when that chain remains inspectable, when exposure of expected answers is controlled, and when the evaluation's conditions remain bound to the governed goal.

The boundary

This is external research on coding-agent refactoring. It does not audit WTK fixtures, measure a defect rate for WTK evaluation, prove that the paper's curation prevents every specification error, or establish that any agent is safe to deploy.

It also does not establish that one benchmark design is the universal remedy. The paper reports its own construction and evaluated conditions. WTK still needs its own repeatable checks before treating a qualification result as evidence for a package or target.

The builder impact

Our take: ask two questions before celebrating a pass. What declared requirement does this check represent? What behavior would make the check fail if an agent merely guessed, copied an answer, or solved a narrower problem than the user asked for?

Those questions make evaluation less glamorous and more useful. They turn a score into a reviewable claim about a named contract. If the answer is unclear, the right result is not extra confidence. It is an explicit limitation and a better fixture.

The WTK test

WTK can evaluate this without importing the benchmark or treating an external score as its own. For a bounded fixture, retain the goal-derived requirement, the acceptance and rejection cases, the expected solution exposure boundary, and the exact target conditions. Have an independent reviewer attempt to identify checks that cannot be traced to a requirement or that accept a prohibited behavior.

Record every planned review result and distinguish fixture validity from the agent's performance on the fixture. A valid-looking pass is not evidence that a test measured the intended outcome until that boundary has been examined.

Still unknown

We do not yet know how often WTK evaluation checks fail this traceability test, whether independent fixture review detects meaningful blind spots, or how validity review affects cost and iteration speed. The next step is a small, provenance-bound fixture audit, not a claim that WTK evaluation has solved benchmark validity.