Math RefereePanel calibration
The first calibration compares Opus 5.5 and Fable 5.1 on ten curator-authored classical tasks: two each for correct proofs with nonobvious steps, false claims with plausible proofs, true statements with defective proofs, underspecified requests, and tasks requiring counterexamples.
There are 60 primary requests: matched high/max effort with 11,000 output tokens, and max effort with equal $0.30 per-model/case allowances. A further 20 exploratory requests used a looser input reservation. These 80 calls concern ten mathematical tasks, not 80 independent problems.
Observations
| Condition | Model | Complete responses | Truth-label matches | Recorded usage |
|---|---|---|---|---|
| Equal dollar limits, max | Fable 5.1 | 8/10 | 8/10 | $2.1037 |
| Equal dollar limits, max | Opus 5.5 | 8/10 | 8/10 | $2.1143 |
| Matched high | Fable 5.1 | 10/10 | 10/10 | $0.7428 |
| Matched high | Opus 5.5 | 10/10 | 10/10 | $0.2763 |
| Matched max | Fable 5.1 | 9/10 | 9/10 | $2.6956 |
| Matched max | Opus 5.5 | 7/10 | 7/10 | $1.7686 |
At high effort both models matched all ten statement-truth labels. This is a classification observation, not a human validation of the proofs. Supporting evidence, issue localization, repairs and appropriate clarification/abstention still need independent grading. A finite-probability counterexample additionally has an exact rational certificate check.
Some max-effort calls exhausted their output caps. Those outcomes have no mathematical verdict to score. Equal maximum dollar allowances do not imply equal actual expenditure. Familiar classical cases, one attempt per condition and a closed-book setup limit conclusions about research mathematics.
Accounting
Recorded usage for all 80 calls is $13.43, against a $25 authorization. This includes billed reasoning/output and token-exhausted attempts, at the published provider rates. It is usage-based accounting, not a provider invoice.
Independent assessment
Download the blind grading package. It contains tasks, anonymous responses and a grading sheet. Model mappings and curator answers are excluded. Human mathematical grading remains pending. No proof-auditing winner is declared.