Updated daily at 05:00 UTC
Benchmark
SimpleQA Verified
Measures: The model answers 1,000 short fact-seeking questions from memory, with no tools. An LLM autorater (gpt-4.1-2025-04-14) grades each answer as correct, incorrect or not attempted. The official headline is F1: the harmonic mean of overall correct and correct-given-attempted.
- Maintainer
- Google DeepMind / Google Research (leaderboard hosted on Kaggle); derived from OpenAI's SimpleQA
- Introduced
- 2025
- Status
- Active
- Saturation
- Headroom
- Typical provenance
- Independent
- In scoring set
- Yes
Leaderboard
Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.
| Model | Accuracy (% correct) | F1 (overall correct x correct-given-attempted, Kaggle leaderboard) | Provenance | Source | Evaluated |
|---|---|---|---|---|---|
| GPT-6 Astra OpenAI · Max effort | 75.6% | — | Independent | Epoch AI | 30 Aug 2026 |
| Gemini 3.1 Pro Google DeepMind · High effort | 73.5% | — | Independent | Epoch AI | 10 Aug 2026 |
| Claude Opus 5.5 Anthropic · Max effort | 72.2% | — | Independent | Epoch AI | 22 Sep 2026 |
| Claude Fable 5.1 Anthropic · Max effort | 70.8% | — | Independent | Epoch AI | 1 Sep 2026 |
| Claude Fable 5 Anthropic · Xhigh effort | 70.7% | — | Independent | Epoch AI | 10 Aug 2026 |
| Gemini 3.8 Flash Google DeepMind · High effort | 69.7% | — | Independent | Epoch AI | 2 Sep 2026 |
| GPT-5.6 Sol OpenAI · Max effort | 69.7% | — | Independent | Epoch AI | 10 Aug 2026 |
| Gemini 3.7 Flash Google DeepMind · High effort | 69.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| Gemini 3 Flash Google DeepMind · High effort | 66.8% | — | Independent | Epoch AI | 27 Aug 2026 |
| Gemini 3.5 Flash Google DeepMind · High effort | 66.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| Gemini 3.6 Flash Google DeepMind · High effort | 66.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.5 OpenAI · Xhigh effort | 63.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| Muse Spark 1.2 Meta · Xhigh effort | 60.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Opus 5 Anthropic · Max effort | 59.9% | — | Independent | Epoch AI | 10 Aug 2026 |
| Muse Spark 1.1 | 57.8% | — | Independent | Epoch AI | 31 Aug 2026 |
| Qwen3.7-Max | 55.8% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Opus 4.8 Anthropic · Max effort | 53.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| DeepSeek-V4-Pro DeepSeek · Max effort | 52.9% | — | Independent | Epoch AI | 27 Aug 2026 |
| Qwen3.6-Max-Preview | 52.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Opus 4.7 Anthropic · Xhigh effort | 51.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| Kimi K3 Moonshot AI · Max effort | 50.6% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5 OpenAI · High effort | 50.1% | — | Independent | Epoch AI | 27 Aug 2026 |
| o3 OpenAI · High effort | 49.4% | — | Independent | Epoch AI | 27 Aug 2026 |
| Grok 4.6 SpaceXAI (xAI) · High effort | 49.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| Grok 4.6 SpaceXAI (xAI) · Xhigh effort | 48.9% | — | Independent | Epoch AI | 27 Aug 2026 |
| Qwen3-Max | 48.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| Grok 4.5 SpaceXAI (xAI) · High effort | 48.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.1 OpenAI · High effort | 48.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| Qwen3.8-Max Alibaba Group · Xhigh effort | 47.3% | — | Independent | Epoch AI | 2 Sep 2026 |
| Claude Opus 4.6 Anthropic · Max effort | 47.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| DeepSeek-V4-Pro DeepSeek · Max effort | 47.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.4 Pro OpenAI · Xhigh effort | 46.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| Qwen3.8-Max Alibaba Group · Xhigh effort | 45.8% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Opus 4.5 Anthropic · 32K effort | 45.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.4 OpenAI · Xhigh effort | 45.1% | — | Independent | Epoch AI | 27 Aug 2026 |
| Qwen3.6-Plus | 44.1% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.6 Terra OpenAI · Max effort | 43.2% | — | Independent | Epoch AI | 10 Aug 2026 |
| o1 OpenAI · High effort | 41.1% | — | Independent | Epoch AI | 31 Aug 2026 |
| GLM-5.3 Z.ai (Zhipu AI) · Max effort | 41.0% | — | Independent | Epoch AI | 28 Aug 2026 |
| GPT-5.6 Luna OpenAI · Max effort | 41.0% | — | Independent | Epoch AI | 10 Aug 2026 |
| Qwen3-235B-A22B-Thinking-2507 | 40.4% | — | Independent | Epoch AI | 27 Aug 2026 |
| Inkling Thinking Machines Lab · Xhigh effort | 40.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.2 OpenAI · Xhigh effort | 37.1% | — | Independent | Epoch AI | 27 Aug 2026 |
| Kimi K2.7 Code | 36.5% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Sonnet 4.6 Anthropic · High effort | 35.5% | — | Independent | Epoch AI | 10 Aug 2026 |
| Kimi K2.6 | 34.9% | — | Independent | Epoch AI | 10 Aug 2026 |
| GPT-5.2 OpenAI · High effort | 34.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| Kimi K2.5 | 34.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| GLM-5.2 Z.ai (Zhipu AI) · Max effort | 34.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| GLM-5.1 | 34.0% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Sonnet 5 Anthropic · Max effort | 33.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| DeepSeek-V4-Flash DeepSeek · Max effort | 33.6% | — | Independent | Epoch AI | 27 Aug 2026 |
| Grok 4.3 SpaceXAI (xAI) · High effort | 33.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Sonnet 5 Anthropic · Xhigh effort | 32.9% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Sonnet 4.6 Anthropic · Max effort | 32.8% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.2 OpenAI · Low effort | 32.8% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.2 OpenAI · Medium effort | 32.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| GLM-4.7 | 32.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-4.1 | 31.1% | — | Independent | Epoch AI | 31 Aug 2026 |
| Claude Sonnet 4.5 Anthropic · 59K effort | 30.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| Grok 4.20 | 30.2% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5.4 mini OpenAI · High effort | 29.4% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-4o | 26.0% | — | Independent | Epoch AI | 31 Aug 2026 |
| Qwen3.5-Plus | 25.4% | — | Independent | Epoch AI | 27 Aug 2026 |
| Claude Sonnet 4.5 | 23.7% | — | Independent | Epoch AI | 10 Aug 2026 |
| GPT-5 mini OpenAI · High effort | 21.6% | — | Independent | Epoch AI | 10 Aug 2026 |
| Qwen3.5-Flash | 20.3% | — | Independent | Epoch AI | 27 Aug 2026 |
| o4-mini OpenAI · Low effort | 19.6% | — | Independent | Epoch AI | 27 Aug 2026 |
| Inkling Small Thinking Machines Lab · Xhigh effort | 19.1% | — | Independent | Epoch AI | 27 Aug 2026 |
| o4-mini OpenAI · High effort | 18.8% | — | Independent | Epoch AI | 27 Aug 2026 |
| Qwen3.6-Flash | 15.9% | — | Independent | Epoch AI | 27 Aug 2026 |
| o3-mini OpenAI · High effort | 15.3% | — | Independent | Epoch AI | 31 Aug 2026 |
| Claude Haiku 4.5 | 13.2% | — | Independent | Epoch AI | 10 Aug 2026 |
| GPT-4.1 mini | 12.7% | — | Independent | Epoch AI | 31 Aug 2026 |
| Claude 3 Opus | 12.6% | — | Independent | Epoch AI | 31 Aug 2026 |
| Claude Haiku 4.5 Anthropic · 32K effort | 12.6% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-5 nano OpenAI · High effort | 11.7% | — | Independent | Epoch AI | 10 Aug 2026 |
| GPT-5.4 nano OpenAI · High effort | 11.7% | — | Independent | Epoch AI | 27 Aug 2026 |
| Gemma 4 31B | 10.4% | — | Independent | Epoch AI | 27 Aug 2026 |
| GPT-4o mini | 8.3% | — | Independent | Epoch AI | 31 Aug 2026 |
| GPT-4.1 nano | 6.0% | — | Independent | Epoch AI | 31 Aug 2026 |
- Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.