Benchmark

GSM8K

Measures: 8,500 grade-school math word problems needing two to eight steps of arithmetic; scored by exact match on the final answer.

Maintainer
OpenAI (Cobbe et al.)
Introduced
2021
Status
Unmaintained
Saturation
Saturated
Typical provenance
Self-reported
In scoring set
No

Paper

Leaderboard

Independent and third-party results only.

Leaderboard
ModelExact matchProvenanceSourceEvaluated
text-babbage-001 (InstructGPT)0.0%Third partyStanford HELM via Epoch AI—
text-ada-001 (InstructGPT)0.4%Third partyStanford HELM via Epoch AI—
GPT-3 ada0.6%Third partyStanford HELM via Epoch AI—
text-curie-001 (InstructGPT)0.6%Third partyStanford HELM via Epoch AI—
GPT-3 babbage0.7%Third partyStanford HELM via Epoch AI—
GPT-3 curie1.6%Third partyStanford HELM via Epoch AI—
OPT-66B1.8%Third partyStanford HELM via Epoch AI—
Falcon-7B Instruct4.0%Third partyStanford HELM via Epoch AI—
Falcon-7B Instruct5.5%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Falcon-7B Instruct6.8%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)6.8%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)8.6%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
GPT-3 davinci9.0%Third partyStanford HELM via Epoch AI—
Llama 2 7B14.6%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
MPT-30B (Instruct, Chat)15.2%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
InternLM-Chat-20B15.7%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Llama 2 7B16.0%Third partyMixtral of Experts via Epoch AI—
Llama 2 7B16.2%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
MPT-30B (Instruct, Chat)16.4%Third partyStanford HELM via Epoch AI—
Llama 2 7B16.7%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Falcon-40B Instruct19.6%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Baichuan2-13B-Chat23.3%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Llama 2 13B24.5%Third partyStanford HELM via Epoch AI—
Falcon-40B Instruct25.0%Third partyStanford HELM via Epoch AI—
Llama 2 13B28.7%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Llama 2 13B28.9%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
ChatGLM2-6B28.9%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Falcon-40B Instruct33.8%Third partyStanford HELM via Epoch AI—
Llama 2 13B34.3%Third partyMixtral of Experts via Epoch AI—
Mistral 7B v0.235.4%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Mistral 7B v0.138.1%Third partyStanford HELM via Epoch AI—
text-davinci-00241.5%Third partyStanford HELM via Epoch AI—
Baichuan2-13B-Chat45.7%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
GPT-3.5 Turbo46.9%Third partyStanford HELM via Epoch AI—
Mistral 7B v0.147.5%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Mistral 7B v0.150.0%Third partyMixtral of Experts via Epoch AI—
text-davinci-00350.6%Third partyStanford HELM via Epoch AI—
Falcon-180B Chat54.4%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Codex (code-davinci-002)56.8%Third partyStanford HELM via Epoch AI—
text-davinci-00357.1%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
GPT-3.5 Turbo57.8%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Qwen-14B-Chat59.5%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Qwen-14B-Chat61.2%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
GPT-490.0%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

GSM8K: leaderboard and what it measures · supermarket.dev