Updated daily at 05:00 UTC
Benchmark
SWE-bench Verified
Measures: An agent in a repository sandbox must write a patch that resolves a real GitHub issue from a popular Python project; scored as the percentage of the 500 human-validated tasks whose hidden fail-to-pass and pass-to-pass tests all pass.
- Maintainer
- SWE-bench team (Princeton/Stanford); OpenAI (Verified subset)
- Introduced
- 2024
- Status
- Superseded
- Saturation
- Saturated
- Typical provenance
- Mixed
- In scoring set
- No
Leaderboard
Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.
| Model | % resolved | Avg cost per instance (bash-only runs on swebench.com) | Provenance | Source | Evaluated |
|---|---|---|---|---|---|
| GPT-4o | 31.0% | — | Independent | Epoch AI | 11 Feb 2026 |
| GPT-4.1 | 48.5% | — | Independent | Epoch AI | 8 Feb 2026 |
| Gemini 2.5 Pro | 57.6% | — | Independent | Epoch AI | 13 Feb 2026 |
| Qwen3.6-Plus | 57.9% | — | Independent | Epoch AI | 14 May 2026 |
| Claude 3.7 Sonnet | 61.0% | — | Independent | Epoch AI | 4 Feb 2026 |
| o3 OpenAI · Medium effort | 62.3% | — | Independent | Epoch AI | 12 Feb 2026 |
| GPT-5 mini OpenAI · Medium effort | 64.7% | — | Independent | Epoch AI | 1 Feb 2026 |
| GPT-5.1 OpenAI · High effort | 65.9% | — | Independent | Epoch AI | 17 Feb 2026 |
| GPT-5.1 OpenAI · High effort | 68.0% | — | Independent | Epoch AI | 18 Feb 2026 |
| Claude Opus 4 | 70.7% | — | Independent | Epoch AI | 6 Feb 2026 |
| Claude Sonnet 4.5 | 71.3% | — | Independent | Epoch AI | 5 Feb 2026 |
| GPT-5 OpenAI · Medium effort | 71.5% | — | Independent | Epoch AI | 5 Feb 2026 |
| GLM-5 | 72.1% | — | Independent | Epoch AI | 15 Feb 2026 |
| Gemini 3 Pro | 72.9% | — | Independent | Epoch AI | 13 Feb 2026 |
| Claude Opus 4.1 | 73.3% | — | Independent | Epoch AI | 11 Feb 2026 |
| GPT-5 OpenAI · High effort | 73.6% | — | Independent | Epoch AI | 6 Feb 2026 |
| GPT-5.2 OpenAI · High effort | 73.8% | — | Independent | Epoch AI | 12 Feb 2026 |
| Kimi K2.5 | 73.8% | — | Independent | Epoch AI | 17 Feb 2026 |
| GLM-5.1 | 74.2% | — | Independent | Epoch AI | 15 May 2026 |
| GPT-5.3-Codex OpenAI · High effort | 74.8% | — | Independent | Epoch AI | 25 Feb 2026 |
| Claude Sonnet 4.6 | 75.2% | — | Independent | Epoch AI | 21 Feb 2026 |
| Gemini 3 Flash | 75.4% | — | Independent | Epoch AI | 18 Feb 2026 |
| Claude Opus 4.6 | 75.6% | — | Independent | Epoch AI | 10 Feb 2026 |
| Gemini 3.1 Pro | 75.6% | — | Independent | Epoch AI | 24 Feb 2026 |
| Claude Opus 4.5 | 76.7% | — | Independent | Epoch AI | 5 Feb 2026 |
| Kimi K2.6 | 76.7% | — | Independent | Epoch AI | 8 May 2026 |
| Qwen3.6-Max-Preview | 76.7% | — | Independent | Epoch AI | 28 May 2026 |
| GPT-5.4 OpenAI · High effort | 76.9% | — | Independent | Epoch AI | 6 Mar 2026 |
| Qwen3.7-Max | 77.3% | — | Independent | Epoch AI | 18 Jun 2026 |
| DeepSeek-V4-Pro DeepSeek · Max effort | 77.6% | — | Independent | Epoch AI | 18 Jun 2026 |
| GLM-5.2 Z.ai (Zhipu AI) · Max effort | 78.7% | — | Independent | Epoch AI | 25 Jun 2026 |
| Claude Opus 4.6 | 78.7% | — | Independent | Epoch AI | 18 Feb 2026 |
| Gemini 3.5 Flash Google DeepMind · High effort | 79.3% | — | Independent | Epoch AI | 1 Jun 2026 |
| Claude Opus 4.7 Anthropic · Max effort | 83.5% | — | Independent | Epoch AI | 20 Apr 2026 |
- Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.