supermarket

Vision-language

o3

By OpenAI, in the o3 family.

Released
16 Apr 2025
Status
Deprecated
Retired
11 Dec 2026
Context
200K
Max output
100K
Input per 1M tokens
$2.00
Output per 1M tokens
$8.00
Open weights
No
License
proprietary
Parameters
—
Knowledge cutoff
Jun 2024
Reasoning
Yes
Modalities
text, image → text

Category scores

Out of 100, from independent and third-party results only, by method category-2026-09-29.1.

Category scores
CategoryScoreRankCoverage
Knowledge & reasoning41.7323 of 4
Coding & software engineering45.1Not ranked2 of 5
Agentic tool use45.2Not ranked1 of 2
Vision & multimodal understanding51.052 of 3
Factuality & hallucination49.4231 of 1
Human preference40.1Not ranked2 of 5

Benchmark results

Every result we hold for this model, with who ran it and where it was published.

Knowledge & reasoning

Knowledge & reasoning results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Chess PuzzlesPuzzles solved38.0%Independentmedium7 Aug 2026Epoch AI
Chess PuzzlesPuzzles solved34.0%Independenthigh7 Aug 2026Epoch AI
Chess PuzzlesPuzzles solved27.0%Independentlow15 Jul 2026Epoch AI
DTBench (Decision Theory Benchmark)Accuracy (capability questions)84.8%Third partyhigh—https://conceptualreasoning.ai/dtbench via Epoch AI
LMCA (Language Model Conceptual Argumentation)Correlation with expert ratings (Pearson r)0.397Third partyhigh—https://conceptualreasoning.ai/lmca via Epoch AI
Mystery Game PuzzlesPuzzles solved29.0%Independenthigh27 Aug 2026Epoch AI
Mystery Game PuzzlesPuzzles solved23.0%Independentmedium27 Aug 2026Epoch AI
SimpleBenchScore (AVG@5)53.1%Third partyhigh—SimpleBench Leaderboard via Epoch AI

Scientific reasoning

Scientific reasoning results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
GPQA DiamondAccuracy81.8%Independenthigh16 Apr 2025Epoch AI
GPQA DiamondAccuracy80.8%Independentmedium7 Aug 2026Epoch AI
GPQA DiamondAccuracy79.8%Independentlow15 Jul 2026Epoch AI

Mathematics

Mathematics results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
FrontierMath Tier 4 (v1)Accuracy2.1%Independenthigh1 Jul 2025Epoch AI
FrontierMath Tiers 1-3Accuracy33.3%Independenthigh27 Aug 2026Epoch AI
FrontierMath Tiers 1-3Accuracy29.8%Independentmedium27 Aug 2026Epoch AI
FrontierMath Tiers 1-3Accuracy19.3%Independentlow27 Aug 2026Epoch AI
FrontierMath Tiers 1-3 (v1)Accuracy18.7%Independenthigh16 Nov 2025Epoch AI
FrontierMath Tiers 1-3 (v1)Accuracy16.9%Independentmedium16 Nov 2025Epoch AI
FrontierMath Tiers 1-3 (v1)Accuracy9.7%Independentlow17 Nov 2025Epoch AI
MATH Level 5Accuracy97.8%Independenthigh16 Apr 2025Epoch AI
OTIS Mock AIME 2024-2025Accuracy84.4%Independentmedium7 Aug 2026Epoch AI
OTIS Mock AIME 2024-2025Accuracy83.9%Independenthigh16 Apr 2025Epoch AI
OTIS Mock AIME 2024-2025Accuracy60.0%Independentlow15 Jul 2026Epoch AI

Coding & software engineering

Coding & software engineering results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Aider PolyglotPercent correct81.3%Third partyhigh—https://aider.chat/docs/leaderboards/ via Epoch AI
Aider PolyglotPercent correct76.9%Third partyunknown—https://aider.chat/docs/leaderboards/ via Epoch AI
Aider PolyglotPercent correct76.9%Third partymedium—Aider LLM Leaderboards via Epoch AI
CadEvalOverall pass rate74.0%Third partymedium—CadEval Dashboard via Epoch AI
GSOOPT@18.8%Third partyhigh—GSO Leaderboard via Epoch AI
METR Time HorizonsAverage task success (Epoch's column)65.4%Third partymedium—METR - Measuring AI Ability to Complete Long Tasks via Epoch AI
METR Time HorizonsAverage task success (Epoch's column)63.6%Source not statedunknown—Epoch AI via Epoch AI
SWE-bench Verified% resolved62.3%Independentmedium12 Feb 2026Epoch AI
WeirdML (v2)Average accuracy52.4%Third partyhigh—https://htihle.github.io/weirdml.html via Epoch AI

Agentic tool use

Agentic tool use results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Deep Research Bench (FutureSearch)Average score45.2%Third partymedium—https://evals.futuresearch.ai/#drb via Epoch AI

Computer & browser use

Computer & browser use results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
OSWorld-Verified% tasks completed23.0%Third partymedium—OS World Website via Epoch AI
OSWorld-Verified% tasks completed17.2%Third partymedium—OS World Website via Epoch AI
OSWorld-Verified% tasks completed9.1%Third partymedium—OS World Website via Epoch AI

Professional knowledge work

Professional knowledge work results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
GDPvalWins vs experts30.8%Source not statedmedium—Epoch AI via Epoch AI

Long context

Long context results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
CL-benchTasks solved17.8%Source not statedhigh—Epoch AI via Epoch AI
Fiction.LiveBench% correct at 16K tokens (Epoch's column)88.9%Third partymedium—Fiction.live leaderboard via Epoch AI

Vision & multimodal understanding

Vision & multimodal understanding results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
GeoBenchCountry accuracy74.0%Third partymedium—GeoBench leaderboard via Epoch AI
GeoBenchCountry accuracy60.0%Third partyhigh—GeoBench leaderboard via Epoch AI
Visual Physics Comprehension Test (VPCT)Correct52.0%Third partymedium—VPCT leaderboard via Epoch AI

Factuality & hallucination

Factuality & hallucination results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
SimpleQA VerifiedAccuracy (% correct)49.4%Independenthigh27 Aug 2026Epoch AI

Writing

Writing results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
LLM Creative Story-Writing Benchmark V1 (Lech Mazur)Mean rubric score (percent of the 10-point scale)83.9%Third partymedium—lechmazur/writing Github repository via Epoch AI

Human preference

Human preference results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Text Arena (arena.ai)Arena score (Elo scale)1432Independent—25 Sep 2026Arena
Text Arena (arena.ai)Votes58587Independent—25 Sep 2026Arena
Vision Arena (arena.ai)Arena score (Elo scale)1216Independent—28 Sep 2026Arena
Vision Arena (arena.ai)Votes45606Independent—28 Sep 2026Arena

Other o3 versions

Sources