Updated daily at 05:00 UTC
- Released
- 4 Nov 2024
- Status
- Retired
- Retired
- 19 Feb 2026
- Context
- 200K
- Max output
- 8.2K
- Input per 1M tokens
- $0.80
- Output per 1M tokens
- $4.00
- Open weights
- No
- License
- proprietary
- Parameters
- —
- Knowledge cutoff
- Jul 2024
- Reasoning
- No
- Modalities
- text, image → text
Category scores
Out of 100, from independent and third-party results only, by method category-2026-09-29.1.
| Category | Score | Rank | Coverage |
|---|---|---|---|
| Coding & software engineering | 28.0 | Not ranked | 1 of 5 |
| Agentic tool use | 19.3 | Not ranked | 1 of 2 |
| Vision & multimodal understanding | 34.0 | Not ranked | 1 of 3 |
| Human preference | 28.3 | Not ranked | 2 of 5 |
Benchmark results
Every result we hold for this model, with who ran it and where it was published.
Knowledge & reasoning
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| DTBench (Decision Theory Benchmark) | Accuracy (capability questions) | 56.7% | Third party | — | — | https://conceptualreasoning.ai/dtbench via Epoch AI |
| MMLU | Accuracy | 74.3% | Third party | — | — | Stanford CRFM Leaderboard via Epoch AI |
Scientific reasoning
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| GPQA Diamond | Accuracy | 38.1% | Independent | — | 12 Mar 2025 | Epoch AI |
Mathematics
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| FrontierMath Tiers 1-3 (v1) | Accuracy | 0.3% | Independent | — | 7 Mar 2025 | Epoch AI |
| MATH Level 5 | Accuracy | 46.4% | Independent | — | 12 Mar 2025 | Epoch AI |
| OTIS Mock AIME 2024-2025 | Accuracy | 4.3% | Independent | — | 25 Feb 2025 | Epoch AI |
Coding & software engineering
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| Aider Polyglot | Percent correct | 28.0% | Third party | — | — | https://aider.chat/docs/leaderboards/ via Epoch AI |
| CadEval | Overall pass rate | 32.0% | Third party | — | — | CadEval Dashboard via Epoch AI |
| WeirdML (v2) | Average accuracy | 30.7% | Third party | — | — | https://htihle.github.io/weirdml.html via Epoch AI |
Agentic tool use
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| BALROG | Average progress | 19.3% | Third party | — | — | https://balrogai.com/ via Epoch AI |
Vision & multimodal understanding
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| GeoBench | Country accuracy | 34.0% | Third party | — | — | GeoBench leaderboard via Epoch AI |
Writing
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| LLM Creative Story-Writing Benchmark V1 (Lech Mazur) | Mean rubric score (percent of the 10-point scale) | 73.5% | Third party | — | — | lechmazur/writing Github repository via Epoch AI |
Human preference
| Benchmark | Metric | Value | Provenance | Setting | Evaluated | Source |
|---|---|---|---|---|---|---|
| Text Arena (arena.ai) | Arena score (Elo scale) | 1325 | Independent | — | 25 Sep 2026 | Arena |
| Text Arena (arena.ai) | Votes | 68875 | Independent | — | 25 Sep 2026 | Arena |
| Vision Arena (arena.ai) | Arena score (Elo scale) | 1127 | Independent | — | 28 Sep 2026 | Arena |
| Vision Arena (arena.ai) | Votes | 1400 | Independent | — | 28 Sep 2026 | Arena |
Other Claude Haiku versions
- Claude Haiku 4.515 Oct 2025
- Claude 3 Haiku13 Mar 2024