DeepSWE

Measures: A coding agent (every model runs on mini-swe-agent, sandboxed, no internet) must complete 113 original, never-released long-horizon engineering tasks across 91 open-source repositories and 5 languages; scored as Pass@1 by hand-written behavioural verifiers run on the committed code in a clean container.

Maintainer: Datacurve
Introduced: 2026
Status: Active
Saturation: Headroom
Typical provenance: Independent
In scoring set: Yes

Paper Official leaderboard

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
Model	Pass@1	Avg cost per task	Avg output tokens per task	Avg agent steps per task	Provenance	Source	Evaluated
GPT-6 Astra OpenAI · Xhigh effort	74.1%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.8 Flash Google DeepMind · High effort	73.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 5 Anthropic · Max effort	73.6%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-6 Astra OpenAI · High effort	73.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-6 Astra OpenAI · Max effort	73.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 5 Anthropic · Xhigh effort	73.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 5 Anthropic · High effort	72.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-6 Astra OpenAI · Medium effort	72.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Sol OpenAI · Max effort	72.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.8 Flash Google DeepMind · Medium effort	71.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Sol OpenAI · Xhigh effort	70.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Fable 5 Anthropic · Xhigh effort	69.9%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Fable 5 Anthropic · Max effort	69.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Terra OpenAI · Max effort	69.6%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Sol OpenAI · High effort	69.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GLM-5.3 Z.ai (Zhipu AI) · Max effort	69.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 5 Anthropic · Medium effort	68.9%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Fable 5 Anthropic · High effort	68.6%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Kimi K3 Moonshot AI · Max effort	68.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Grok 4.6 SpaceXAI (xAI) · Medium effort	67.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Luna OpenAI · Max effort	67.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.5 OpenAI · Xhigh effort	67.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-6 Astra OpenAI · Low effort	67.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Grok 4.6 SpaceXAI (xAI) · Xhigh effort	66.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.7 Flash Google DeepMind · Medium effort	65.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Fable 5 Anthropic · Medium effort	65.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.7 Flash Google DeepMind · High effort	65.3%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Grok 4.6 SpaceXAI (xAI) · High effort	65.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.5 OpenAI · High effort	64.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GLM-5.3-Flash Z.ai (Zhipu AI) · Max effort	63.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Sol OpenAI · Medium effort	61.1%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Terra OpenAI · Xhigh effort	60.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Fable 5 Anthropic · Low effort	59.6%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 4.8 Anthropic · Max effort	59.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 5 Anthropic · Low effort	58.1%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Qwen3.8-Max Alibaba Group · Xhigh effort	57.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Luna OpenAI · Xhigh effort	56.9%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Muse Spark 1.2 Meta · Xhigh effort	54.9%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 4.8 Anthropic · Xhigh effort	54.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.5 OpenAI · Medium effort	54.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Sonnet 5 Anthropic · Max effort	53.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.7 Flash Google DeepMind · Low effort	53.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Terra OpenAI · High effort	53.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Grok 4.5 SpaceXAI (xAI) · High effort	53.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Muse Spark 1.1 Meta	53.3%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 4.8 Anthropic · High effort	51.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.4 OpenAI · Xhigh effort	51.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Sonnet 5 Anthropic · Xhigh effort	49.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 4.8 Anthropic · Medium effort	48.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Sonnet 5 Anthropic · High effort	48.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.6 Flash Google DeepMind · High effort	46.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Sol OpenAI · Low effort	45.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Luna OpenAI · High effort	44.2%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GLM-5.2 Z.ai (Zhipu AI) · Max effort	43.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Grok 4.6 SpaceXAI (xAI) · Low effort	41.6%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Opus 4.8 Anthropic · Low effort	40.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Sonnet 5 Anthropic · Medium effort	39.8%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.5 Flash Google DeepMind · Medium effort	37.4%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GLM-5.2 Z.ai (Zhipu AI) · High effort	36.3%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.5 Flash Google DeepMind · High effort	36.1%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Terra OpenAI · Medium effort	35.1%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Kimi K2.7 Code Moonshot AI	30.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Sonnet 5 Anthropic · Low effort	30.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Claude Sonnet 4.6 Anthropic · High effort	29.9%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.5 OpenAI · Low effort	27.0%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Terra OpenAI · Low effort	24.1%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
Gemini 3.1 Pro Google DeepMind	11.7%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Luna OpenAI · Medium effort	11.3%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—
GPT-5.6 Luna OpenAI · Low effort	1.5%	—	—	—	Third party	https://deepswe.datacurve.ai/ via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI