Benchmarks

Broad expert knowledge and multi-step reasoning on hard, closed-answer questions across many fields.

• AI2 Reasoning Challenge (ARC)UnmaintainedSaturated62 results
• ARC-AGI-1ActiveSaturated0 results
• ARC-AGI-2ActiveSaturated0 results
• ARC-AGI-3ActiveNear-ceilingCounts toward the score0 results
• Adversarial NLI (ANLI)UnmaintainedNear-ceiling9 results
• BIG-Bench Hard (BBH)UnmaintainedNear-ceiling44 results
• Chess PuzzlesActiveHeadroomCounts toward the score213 results
• DTBench (Decision Theory Benchmark)ActiveSaturated208 results
• Earthborne Rangers (EBR-bench)ActiveHeadroomCounts toward the score23 results
• EnigmaEvalActiveHeadroom0 results
• HLE-DiamondActiveHeadroom0 results
• HellaSwagUnmaintainedSaturated63 results
• Humanity's Last Exam (HLE)ActiveHeadroomCounts toward the score46 results
• LAMBADAUnmaintainedNear-ceiling20 results
• LMCA (Language Model Conceptual Argumentation)ActiveNear-ceilingCounts toward the score172 results
• LiveBenchActiveNear-ceilingCounts toward the score0 results
• MMLUUnmaintainedNear-ceiling133 results
• MMLU-ProUnmaintainedSaturated71 results
• Mystery Game PuzzlesActiveNear-ceiling128 results
• OpenBookQAUnmaintainedNear-ceiling34 results
• PIQAUnmaintainedNear-ceiling57 results
• ScienceQAUnmaintainedSaturated11 results
• SimpleBenchActiveNear-ceilingCounts toward the score100 results
• SuperGLUEUnmaintainedNear-ceiling1 result
• WinoGrandeUnmaintainedHeadroom66 results

Graduate- and research-level physics, chemistry and biology problems, including scientific coding.

• CritPtActiveHeadroomCounts toward the score0 results
• FrontierScienceActiveHeadroom0 results
• GPQA DiamondActiveSaturated358 results
• SciCodeActiveHeadroomCounts toward the score0 results
• Surface Evolver BenchActiveSaturated28 results

Competition and research mathematics, scored on final answers or on graded proofs.

• AIME 2025/2026 (MathArena)SupersededSaturated0 results
• FrontierMath ErdősActiveHeadroomCounts toward the score0 results
• FrontierMath Tier 4ActiveSaturated63 results
• FrontierMath Tier 4 (v1)SupersededHeadroom61 results
• FrontierMath Tiers 1-3ActiveSaturated108 results
• FrontierMath Tiers 1-3 (v1)SupersededSaturated94 results
• GSM8KUnmaintainedSaturated88 results
• IMO-ProofBench (Advanced)ActiveSaturated0 results
• MATH Level 5ActiveSaturated106 results
• MathArena (ArXivMath, BrokenArXiv, ArXivLean)ActiveNear-ceilingCounts toward the score0 results
• OTIS Mock AIME 2024-2025ActiveSaturated281 results
• ProofBench (Vals AI, FormalProofBench)ActiveSaturated0 results

Writing, fixing and running code in real repositories and terminals, judged by hidden tests.

• Aider PolyglotActiveNear-ceilingCounts toward the score67 results
• CadEvalUnmaintainedHeadroom14 results
• CursorBenchActiveHeadroomCounts toward the score0 results
• CybenchActiveSaturated21 results
• DeepSWEActiveHeadroomCounts toward the score69 results
• ExploitBenchActiveHeadroom18 results
• FrontierCodeActiveHeadroomCounts toward the score34 results
• FrontierSWEActiveHeadroomCounts toward the score0 results
• GSOActiveHeadroomCounts toward the score37 results
• METR Time HorizonsActiveNear-ceiling0 results
• MirrorCodeActiveHeadroomCounts toward the score9 results
• PostTrainBenchActiveHeadroom12 results
• SWE-bench ProActiveSaturated0 results
• SWE-bench VerifiedSupersededSaturated34 results
• SWE-rebenchActiveHeadroomCounts toward the score0 results
• Terminal-BenchActiveHeadroomCounts toward the score0 results
• Terminal-Bench 2.0SupersededNear-ceiling198 results
• WeirdML (v2)ActiveSaturated166 results

Calling functions and MCP tools correctly over many turns, following policies, researching answers with search tools, and running long tasks on its own.

• AutomationBench (and AutomationBench-AA)ActiveHeadroomCounts toward the score0 results
• BALROGActiveHeadroomCounts toward the score41 results
• BrowseCompActiveSaturated0 results
• Deep Research Bench (FutureSearch)ActiveHeadroomCounts toward the score41 results
• EnterpriseOps-GymActiveHeadroomCounts toward the score0 results
• MCP-AtlasActiveNear-ceilingCounts toward the score0 results
• Toolathlon (Tool Decathlon)ActiveHeadroomCounts toward the score0 results
• Vending-Bench 2ActiveHeadroom0 results
• τ²-bench (tau2-bench)SupersededSaturated0 results
• τ³-bench (tau3-bench)ActiveHeadroomCounts toward the score0 results

Operating a desktop, phone or web browser from screenshots or accessibility data to finish multi-step tasks.

• AndroidWorldActiveSaturated0 results
• OSWorld 2.0ActiveHeadroomCounts toward the score16 results
• OSWorld-VerifiedSupersededSaturated20 results
• ScreenSpot-ProActiveSaturated0 results

Producing real deliverables (documents, spreadsheets, analyses) of the kind professionals are paid for, judged against expert work.

• AA-BriefcaseActiveHeadroomCounts toward the score0 results
• APEX-AgentsActiveHeadroomCounts toward the score38 results
• GDPvalSupersededHeadroom0 results
• GDPval-AAActiveHeadroomCounts toward the score0 results
• Remote Labor Index (RLI)ActiveHeadroomCounts toward the score0 results
• TheAgentCompanyActiveHeadroomCounts toward the score16 results

Finding, tracking and reasoning over information spread across very long inputs (100K to 1M+ tokens).

• AA-LCR (Artificial Analysis Long Context Reasoning)ActiveNear-ceilingCounts toward the score0 results
• CL-benchActiveHeadroom23 results
• CL-bench LifeActiveHeadroom17 results
• Fiction.LiveBenchUnmaintainedSaturated0 results
• GraphWalksActiveHeadroom0 results
• LongBench v2ActiveHeadroom0 results
• OpenAI MRCR v2 (8-needle)ActiveNear-ceiling0 results

Reasoning over images, charts, diagrams and video together with text.

• CharXiv (Reasoning)ActiveSaturated0 results
• Furniture AssemblyActiveHeadroomCounts toward the score27 results
• GeoBenchActiveNear-ceilingCounts toward the score32 results
• MMMUSupersededNear-ceiling0 results
• MMMU-ProActiveNear-ceilingCounts toward the score0 results
• Video-MMMUActiveNear-ceiling0 results
• Visual Physics Comprehension Test (VPCT)ActiveNear-ceilingCounts toward the score38 results
• ZeroBenchActiveHeadroomCounts toward the score0 results

Reading text, tables, formulas and layout from scanned or born-digital documents and turning them into structured output.

• DocVQAActiveSaturated0 results
• GDP.pdfActiveHeadroomCounts toward the score0 results
• IDP Leaderboard (Intelligent Document Processing)ActiveHeadroom0 results
• OCRBench v2ActiveHeadroomCounts toward the score0 results
• OmniDocBenchActiveSaturated0 results
• olmOCR-BenchActiveHeadroomCounts toward the score0 results

Legal reasoning, research and drafting judged against lawyers' answers or expert rubrics.

• Harvey Legal Agent Benchmark (LAB)ActiveHeadroomCounts toward the score0 results
• Legal Research BenchActiveHeadroomCounts toward the score0 results
• LegalBenchActiveSaturated0 results
• PRBench Legal (Professional Reasoning Benchmark)ActiveHeadroomCounts toward the score0 results

Clinical knowledge and safe, useful health conversations judged against physician answers or rubrics.

• HealthBench (incl. Hard and Consensus)ActiveHeadroom0 results
• HealthBench ProfessionalActiveHeadroom0 results
• MLCR (Medical Long Context Reasoning) / MLCR-AAActiveHeadroomCounts toward the score0 results
• MedCodeActiveHeadroomCounts toward the score0 results
• MedHELMActive0 results
• MedQA (USMLE)UnmaintainedSaturated0 results
• MedScribeActiveNear-ceilingCounts toward the score0 results

Financial analysis, filings research, accounting and tax questions, often with tools and real documents.

• Excel Modeling Benchmark (EMB)ActiveHeadroomCounts toward the score0 results
• Finance Agent v1.1SupersededHeadroom0 results
• Finance Agent v2ActiveHeadroomCounts toward the score0 results
• PRBench Finance (Professional Reasoning Benchmark)ActiveHeadroomCounts toward the score0 results
• Tax Agent BenchActiveHeadroomCounts toward the score0 results

Knowledge, reasoning and translation quality in languages other than English, including low-resource ones.

• Global-MMLU-LiteActiveSaturated0 results
• MMLU-ProXActiveNear-ceiling0 results
• MMMLU (Multilingual MMLU)ActiveSaturated0 results
• MultiNRC (Multilingual Native Reasoning Challenge)ActiveHeadroomCounts toward the score0 results
• WMT24++ActiveNear-ceiling0 results

Answering factual questions correctly, abstaining when unsure, and staying grounded in provided sources.

• AA-OmniscienceActiveHeadroomCounts toward the score0 results
• FACTS Benchmark SuiteActiveHeadroomCounts toward the score0 results
• FACTS GroundingActiveHeadroom0 results
• SimpleQASupersededHeadroom0 results
• SimpleQA VerifiedActiveHeadroomCounts toward the score81 results
• TriviaQAUnmaintainedNear-ceiling53 results
• Vectara Hallucination Leaderboard (HHEM)ActiveNear-ceilingCounts toward the score0 results

Obeying precise, verifiable constraints and holding instructions across multi-turn conversations.

• IFBenchActiveNear-ceilingCounts toward the score0 results
• IFEvalSupersededSaturated0 results
• MultiChallengeActiveHeadroomCounts toward the score0 results

Quality of creative and long-form fiction, judged by LLM graders with rubrics and pairwise matchups.

• EQ-Bench Creative Writing v3ActiveHeadroomCounts toward the score0 results
• EQ-Bench Longform Creative WritingActiveNear-ceilingCounts toward the score0 results
• LLM Creative Story-Writing Benchmark (Lech Mazur)ActiveHeadroomCounts toward the score0 results
• LLM Creative Story-Writing Benchmark V1 (Lech Mazur)SupersededNear-ceiling45 results
• WritingBenchUnmaintainedNear-ceiling0 results

Transcribing speech accurately, and understanding or reasoning over spoken audio, sound and music; voice quality is ranked under human preference.

• Artificial Analysis Speech to Text (AA-WER)ActiveNear-ceilingCounts toward the score0 results
• Big Bench AudioActiveSaturated0 results
• MMAU-ProActiveHeadroom0 results
• Open ASR LeaderboardActiveNear-ceilingCounts toward the score0 results
• τ-Voice (tau-Voice)ActiveHeadroomCounts toward the score0 results

Which model people prefer in blind side-by-side votes (chat, web dev, vision, search, documents, agents, voices), reported as Elo with confidence intervals.

• Agent Arena (arena.ai)ActiveHeadroomCounts toward the score0 results
• Artificial Analysis Speech Agent ArenaActiveHeadroomCounts toward the score0 results
• Artificial Analysis Text to Speech ArenaActiveHeadroomCounts toward the score0 results
• Code Arena: WebDev (arena.ai)ActiveHeadroomCounts toward the score109 results
• Document Arena (arena.ai)ActiveHeadroomCounts toward the score42 results
• LMArena Text: Creative Writing categoryActiveHeadroom0 results
• LMArena Text: Instruction Following categoryActiveHeadroom0 results
• Search Arena (arena.ai)ActiveHeadroomCounts toward the score8 results
• Text Arena (arena.ai)ActiveHeadroomCounts toward the score297 results
• Vision Arena (arena.ai)ActiveHeadroomCounts toward the score129 results