All 21 models from the build, one question at a time: three sizes (125M, 500M, Gemma 2B) by seven training stages. An LLM judge scores every response out of 10, and for every held-out question the judge is handed the gold answer and its corpus evidence, so scores are checkable rather than vibes.
Every stage exists in all three sizes: 125M and 500M trained from scratch, plus Gemma 2 2B. Click a size to open that model's own site (each site carries the model's training details, cost, and HuggingFace link).
Loading baked results...
| # | Model | Stage | Mean /10 | QA | Instr | RAFT | Refusal | n |
|---|
Click a model to see its per-question scores; click a question to read the response, the gold answer, and the judge's reason.
These models were trained on a specific legal and financial corpus. A judge model has never read that corpus, so asking it "is this answer right?" invites two failure modes: it grades corpus facts against its own general knowledge, and nobody can audit the result without reading the corpus themselves.
We built a held-out evaluation set directly from the corpus. Gemini 2.5 Pro read fresh corpus passages (sampled with a different seed than any training data, then filtered so no question overlaps the QA, instruction, preference, or RAFT fine-tuning sets by even a 6-word phrase) and authored for each item: a self-contained question, a gold answer, and a verbatim evidence quote from the passage.
The judge (Gemini 2.5 Flash, temperature 0, a different model family than anything being judged) then grades each response blind and one at a time: it sees the question, the gold answer, the evidence, and the response. It never sees which model answered, and it never has to recall a fact: its only job is comparison.
| Dimension | Points | What it measures |
|---|---|---|
| Correctness | 0-5 | Factual agreement with the gold answer |
| Completeness | 0-2 | Covers the gold's key points, not just one |
| Groundedness | 0-2 | No invented cases, figures, or citations |
| Clarity | 0-1 | Answers what was asked, obeys format constraints |
Two rules are baked in: a confidently wrong answer must score below an honest "I don't know", and partially correct content earns its correctness points even when incomplete.
The three base models are raw text completers, not chat assistants. They get a two-example few-shot prompt instead of a bare question, their run-on text is trimmed at the next "Question:", and the judge is told to grade content only. With partial credit, a relevant but rambling completion lands at 2-4 rather than 0; and when a zero does appear, the stored gold answer and evidence make it verifiable.
Closed-book QA tests knowledge in the weights. Instruction items attach a TEXT block and a task (summarize, extract, rewrite, classify, draft, format-constrained). RAFT items provide one golden passage among three distractors; refusal items provide only distractors, and the gold behavior is to say the context does not contain the answer. Every model answers every item, shaped the way that model was trained to consume input, so RAFT models really are tested closed-book on QA items (they usually refuse: that is what they were trained to do, and the per-task breakdown on the leaderboard shows it).
The arena also takes free-form questions. Those have no gold answer, so the judge falls back to writing its own reference first. The UI labels these scores as unaudited; they never mix into the leaderboard.