Benchmark

OpenBookQA

Measures: Four-option elementary science questions that need one core science fact from a small 'book' combined with broad common knowledge.

Maintainer
Allen Institute for AI (Mihaylov et al.)
Introduced
2018
Status
Unmaintained
Saturation
Near-ceiling
Typical provenance
Self-reported
In scoring set
No

Paper

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

Leaderboard
ModelAccuracyProvenanceSourceEvaluated
GPT-2 XL (1.5B)22.4%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
OPT-1.3B24.0%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
Llama 2 7B31.4%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)31.4%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
Falcon-7B Instruct32.0%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
StableLM-Alpha 7B32.4%Self-reportedXGen-7B Technical Report via Epoch AI—
Phi-1.5
Microsoft · 5 effort
37.2%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
Dolly 2.0 (3B, 7B, 12B)39.2%Self-reportedXGen-7B Technical Report via Epoch AI—
OPT-13B39.8%Self-reportedXGen-7B Technical Report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)41.8%Self-reportedXGen-7B Technical Report via Epoch AI—
Falcon-7B Instruct44.0%Self-reportedXGen-7B Technical Report via Epoch AI—
Llama 3 8B45.0%Self-reportedThe Llama 3 Herd of Models via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)51.4%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Falcon-7B Instruct51.6%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
MPT-30B (Instruct, Chat)52.0%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Falcon-40B Instruct56.6%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Llama 2 13B57.0%Self-reportedLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
text-davinci-001 (InstructGPT)57.6%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)57.6%Third partyLanguage Models are Few-Shot Learners via Epoch AI—
text-davinci-001 (InstructGPT)57.6%Third partyLLaMA: Open and Efficient Foundation Language Models via Epoch AI—
Llama 2 7B58.6%Self-reportedLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
text-davinci-001 (InstructGPT)58.8%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)58.8%Third partyLanguage Models are Few-Shot Learners via Epoch AI—
Falcon-180B Chat64.2%Self-reportedThe Falcon Series of Open Language Models via Epoch AI—
text-davinci-001 (InstructGPT)65.4%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)65.4%Third partyLanguage Models are Few-Shot Learners via Epoch AI—
Phi-273.6%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Phi-273.6%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Gemma 7B78.6%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Mistral 7B v0.179.8%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Llama 3 8B82.6%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
GPT-3.5 Turbo86.0%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Phi-3-medium87.4%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Phi-3-medium87.4%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

OpenBookQA: leaderboard and what it measures · supermarket.dev