A Judge Can Become Code Without Becoming Truth

An LLM judge does not have to remain an opaque model call. PAJAMA asks a model to write the judging logic once, then runs that logic as a committee of ordinary Python programs.

That is a genuinely useful shift. The programs can be inspected, versioned, tested, and patched. They are also fast. Across five pairwise-preference datasets, PAJAMA's selected committees averaged 78.11 percent accuracy and 439.41 samples per second. That was close to OLMo-2-13B-Instruct's 79.70 percent average accuracy at 9.30 samples per second, a 47.25-fold throughput difference.

The method is more structured than simply asking a model to emit a score. Claude Opus 4.6 synthesized 80 candidate programs from ten curated rubrics. PAJAMA calibrated abstention thresholds on labeled validation examples, removed programs below chance, selected the strongest remaining programs, and learned how to aggregate their votes. When the committee was uncovered or uncertain, it could route the example back to an LLM judge.

That makes execution deterministic once the programs, thresholds, and aggregation model are fixed. It does not make the verdict ground truth. The rubrics were chosen by people, the programs were generated by a model, and selection depended on labels that were often produced by other models. The study also excluded coding and mathematics prompts, removed ambiguous or low-confidence preference cases, and randomly labeled the small uncovered remainder in its standalone comparison. Its own limitations say the method is best suited to tasks with relatively straightforward evaluation.

This is where PAJAMA and WTK's tribunal approach rhyme without becoming the same design. WTK separates deterministic evidence from advisory semantic judgment. A receipt, schema violation, or missing required artifact does not become negotiable because a panel likes the answer. PAJAMA instead turns some semantic heuristics into deterministic program execution, then combines their noisy outputs probabilistically. One protects authority boundaries. The other makes a portion of judgment cheaper, faster, and easier to inspect.

Our take: program distillation is a strong candidate for a tribunal lens or a first-pass evaluator, not a substitute for the tribunal's evidence hierarchy. A WTK version would bind the synthesized programs and calibration data to exact digests, sandbox their execution, retain abstentions and disagreement, test them on held-out counterexamples, and prevent a favorable aggregate from overriding deterministic failures. Program patches would create a new evaluator version and require new evidence.

WTK could test that claim with a predeclared three-arm comparison: a conventional LLM judge, a PAJAMA-style compiled program committee, and a tribunal combining deterministic gates with programmatic judging and LLM fallback. Each arm would evaluate the same held-out evidence under fixed attempt counts. We would compare false passes, false failures, abstentions, disagreement, calibration, cost, latency, and sensitivity to adversarial formatting. Then we would patch one known program weakness and rerun the held-out set under a new evaluator identity. The question is not whether compiled judges are deterministic. It is whether they add reliable, inspectable judgment without weakening deterministic evidence authority.

The bias experiment makes that direction especially interesting. PAJAMA used a coding agent to edit the visible scoring logic after detecting superficial preferences. Average flip and bias-win measures improved, although one rich-content bias-win measure worsened. Inspectability made repair possible, but the mixed result is a reminder that a patch must be requalified rather than trusted because it is readable.

One correction to our own record matters. An earlier WTK note described PAJAMA as matching a 13B judge "at a hundredth of the cost." The paper does not support that combined claim. It reports the 47.25-fold throughput comparison above. Separately, reward models trained from PAJAMA labels used 45 to 50 times less proprietary-labeling API cost than the GPT-4 labeling comparison. Those results are promising, bounded, and different.

The useful lesson is not that judges have become deterministic. It is that some judgment can be compiled into inspectable machinery, while uncertainty and evidence authority remain explicit. That is exactly the kind of component worth testing inside a governed evaluation system.