← Home

Learning Strategies to Break Judges

Guru Shabadi, Aaditya Naik, Rajeev Alur, Mayur Naik

Read the preprint

OpenAI withdrew three AI-generated math manuscripts after a sign error in one undermined the other two, an example of why we need stronger proof judges.

In this work, we were able to distill interpretable strategies to introduce logical errors in math proofs that even frontier agentic judges were not able to detect, including GPT 5.6 and Opus 5 at their highest reasoning budgets with their Codex and Claude Code harnesses. This is an impressive feat because (1) these judges had open access to the proof files and freedom to pick their verification strategies, and (2) frontier AI systems have been used to resolve several open problems in mathematics and theoretical computer science.

Our pipeline consists of a discovery phase, where agents make repeated attempts to inject errors into correct proofs followed by which a distillation agent extracts effective strategies from these traces. We then show that these strategies help agents produce logical errors that are able to bypass judges zero-shot.

Pipeline from proof corpora through discovery and distillation to evaluation on held-out proofs.
During discovery, mutator agents inject errors into sound proofs and receive feedback from agentic judges. A distillation agent turns their attempts into a strategy guide. On held-out proofs, judges miss errors more often when mutators use the guide.

We apply our method to multiple datasets: graduate math lecture notes, journal papers, and the set of proofs addressing ten open problems in mathematics and theoretical computer science published by OpenAI this summer. Here are examples of two successful strategies:

Dropping a ceiling in a relative asymptotic

In a proof from the OpenAI theoretical computer science dataset, a remainder bound contains the term eyyp/Γ(1 + p), with p = ⌈L⌉ + ½ and L = log λ.

Original proof
Uses Stirling’s formula to bound the logarithm of the term with the ceiling in place.
Mutation
Replaces ⌈L⌉ with L and claims the term changes only by a factor tending to one.

Along a sequence where the rounding error approaches ½ and y = L/8, the ratio of the original term to the unrounded term tends to 1/√8. The logarithmic bound still holds, but the inserted relative approximation is false.

Six of ten successful mutations in the GPT-5.6-sol evaluation on this dataset used this kind of change.

Weakening the definition of boundary slope

A knot theory proof uses two conditions on the surface that realizes a boundary slope.

Original definition
The surface is incompressible and boundary-incompressible.
Mutation
The surface is incompressible and has no closed components.

Take an annular neighborhood of a peripheral slope on the boundary of the knot exterior and push its interior slightly into the exterior. The resulting annulus meets the altered requirements and is boundary-compressible. This construction makes every peripheral slope a boundary slope under the altered definition, so the later argument can no longer invoke the finiteness theorem for boundary slopes.

All three Claude Opus 5 judges missed this mutation.

Experimental results

In the held-out evaluation, the judge miss rate is the number of completed judge reviews counted as missing the planted error divided by the total number of completed judge reviews of valid mutations. Each valid mutation was assigned three independent judges, and each completed review contributes separately to the rate.

  • Learned strategies transferred to unseen proofs. Giving each of the four datasets equal weight, the average ratio of guided to unguided judge miss rates was 1.5× for GPT-5.6-sol and 3.5× for Claude Opus 5.
  • The gains varied by dataset. Guidance increased miss rates for Claude Opus 5 on all four datasets. For GPT-5.6-sol, the largest gain was on OpenAI-TCS. Olympiad proofs had the lowest held-out miss rates for both models. We conjecture that this is due to the increasing complexity of the proofs.
GPT-5.6-sol miss rates during discovery and held-out evaluation. The largest guided gain is on OpenAI-TCS.
GPT-5.6-sol · Discovery and held-out miss rates
Claude Opus 5 miss rates during discovery and held-out evaluation. Guided miss rates exceed unguided rates on all four datasets.
Claude Opus 5 · Discovery and held-out miss rates
  • Some errors fooled both model families. When the team gave the other model the errors most often missed in the original runs, some still went undetected, especially on OpenAI-TCS.
  • Misses persisted at the highest reasoning settings. On selected errors previously missed at medium effort, GPT-5.6-sol (ultra) missed 7 of 12 and Claude Opus 5 (max) recorded 11 misses in 11 evaluations.
Miss rates when each model judged selected proof errors generated against the other model.
Cross-model judging on selected errors
At the highest reasoning settings, GPT-5.6-sol missed 7 of 12 selected errors and Claude Opus 5 recorded 11 misses in 11 evaluations.
Judging at higher reasoning effort on previously missed errors

See Section 3.2 and Figures 3–4 for the evaluation setup and full breakdown. Read the preprint.