MMLU

Measures: Four-option multiple-choice exam questions across 57 subjects, from elementary mathematics to law and medicine.

Maintainer: Hendrycks et al.
Introduced: 2020
Status: Unmaintained
Saturation: Near-ceiling
Typical provenance: Self-reported
In scoring set: No

Paper

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
Model	Accuracy	Provenance	Source	Evaluated
Claude 3.5 Sonnet Anthropic	87.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
DeepSeek-V3 DeepSeek	87.2%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemini 1.5 Pro Google DeepMind	86.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Claude 3.5 Sonnet Anthropic	86.5%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemini 1.5 Pro Google DeepMind	85.9%	Third party	Gemini 1.5 Report via Epoch AI	—
Claude 3 Opus Anthropic	84.6%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3.1 405B Meta	84.5%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-4o OpenAI	84.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-4o OpenAI	84.2%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Qwen2.5-72B-Instruct Alibaba Group	83.4%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-4 OpenAI	82.4%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Qwen2-72B-Instruct Alibaba Group	82.4%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Amazon Nova Pro Amazon	82.0%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-4 Turbo OpenAI	81.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3.2 90B Vision Meta	80.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3.1 70B Meta	80.1%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Mistral Large 2.0 Mistral AI	80.0%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemini 2.0 Flash Google DeepMind	79.7%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-4 Turbo OpenAI	79.6%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3 70B Meta	79.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Yi-Large 01.AI	79.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3.3 70B Meta	79.1%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemini 1.5 Flash Google DeepMind	77.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Phi-3-medium Microsoft	77.5%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Amazon Nova Lite Amazon	77.0%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-4o mini OpenAI	76.7%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Claude 3 Sonnet Anthropic	75.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemma 2 27B Google DeepMind	75.7%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Claude 3.5 Haiku Anthropic	74.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemini 1.5 Flash Google DeepMind	73.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Claude 3 Haiku Anthropic	73.8%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Claude 2.1 Anthropic	73.5%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Qwen2.5-7B-Instruct Alibaba Group	72.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemma 2 9B Google DeepMind	72.1%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Amazon Nova Micro Amazon	70.8%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemini 1.0 Pro Google DeepMind	70.0%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Command R+ (08-2024) Cohere	69.4%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-3.5 Turbo OpenAI	68.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Claude Instant 1.2 Anthropic	68.8%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3 8B Meta	68.8%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Mistral Large 1.0 Mistral AI	68.8%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Mistral Small 1.0 Mistral AI	68.7%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
GPT-3.5 Turbo OpenAI	67.3%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Gemma 7B Google DeepMind	66.1%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Command R (08-2024) Cohere	65.2%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Qwen-14B-Chat Alibaba Group	65.0%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Qwen-14B-Chat Alibaba Group	64.0%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Mistral 7B v0.1 Mistral AI	62.5%	Third party	Mixtral of Experts via Epoch AI	—
Mistral 7B v0.2 Mistral AI	62.5%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
Mistral 7B v0.3 Mistral AI	59.9%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Phi-2 Microsoft	58.4%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
InternLM-Chat-20B Shanghai AI Laboratory	57.4%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Mistral 7B v0.1 Mistral AI	56.6%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3.2 11B Vision Meta	56.5%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 3.1 8B Meta	56.1%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
InternLM-Chat-20B Shanghai AI Laboratory	55.6%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Llama 2 13B Meta	55.6%	Third party	Mixtral of Experts via Epoch AI	—
Falcon-40B Instruct Technology Innovation Institute (TII)	55.4%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Llama 2 13B Meta	55.4%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Baichuan2-13B-Chat Baichuan Intelligence	55.1%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Llama 2 13B Meta	55.1%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
Llama 2 13B Meta	54.8%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
Baichuan2-13B-Chat Baichuan Intelligence	50.1%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
MPT-30B (Instruct, Chat) Databricks	46.9%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Llama 2 7B Meta	45.8%	Third party	Stanford CRFM Leaderboard via Epoch AI	—
Llama 2 7B Meta	45.7%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
Llama 2 7B Meta	45.3%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
Llama 2 7B Meta	44.4%	Third party	Mixtral of Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	43.9%	Third party	Scaling Language Models: Methods, Analysis & Insights from Training Gopher via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	43.9%	Third party	LLaMA: Open and Efficient Foundation Language Models via Epoch AI	—
MPT-7B (Instruct, Chat, StoryWriter) Databricks	27.9%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
MPT-7B (Instruct, Chat, StoryWriter) Databricks	26.8%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Falcon-7B Instruct Technology Innovation Institute (TII)	26.2%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Falcon-7B Instruct Technology Innovation Institute (TII)	26.0%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI