Benchmark

BIG-Bench Hard (BBH)

Measures: 23 BIG-bench tasks on which language models had trailed average human raters, probing multi-step symbolic, logical and arithmetic reasoning.

Maintainer
Suzgun et al. (Google Research and Stanford)
Introduced
2022
Status
Unmaintained
Saturation
Near-ceiling
Typical provenance
Self-reported
In scoring set
No

Paper

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

Leaderboard
ModelAverage accuracyProvenanceSourceEvaluated
Gemini 1.5 Pro89.2%Third partyGemini 1.5 report via Epoch AI—
DeepSeek-V387.5%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
DeepSeek-V382.9%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
Llama 3.1 405B82.9%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
Llama 3.1 405B82.9%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
Phi-3-medium81.4%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
DeepSeek-V278.8%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
DeepSeek-V278.8%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
GPT-475.1%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Yi-34B-Chat71.7%Self-reportedYi: Open Foundation Models by 01.AI via Epoch AI—
Stable Beluga 2 (Llama 2 70B fine-tune)69.3%Self-reportedQwen Technical Report via Epoch AI—
GPT-3.5 Turbo61.6%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Phi-259.4%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Mistral 7B v0.156.1%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Gemma 7B55.1%Self-reportedGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Gemma 7B55.1%Self-reportedNemotron-4 15B Technical Report via Epoch AI—
Qwen-14B-Chat55.0%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Yi-34B-Chat51.4%Self-reportedYi: Open Foundation Models by 01.AI via Epoch AI—
Qwen-14B-Chat49.7%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Baichuan2-13B-Chat47.2%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Yi-6B-Chat47.2%Self-reportedYi: Open Foundation Models by 01.AI via Epoch AI—
Llama 2 13B47.0%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
InternLM-Chat-20B42.4%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
Yi-6B-Chat39.7%Self-reportedYi: Open Foundation Models by 01.AI via Epoch AI—
Mistral 7B v0.139.5%Self-reportedNemotron-4 15B Technical Report via Epoch AI—
Llama 2 13B39.4%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Llama 2 13B39.4%Self-reportedNemotron-4 15B Technical Report via Epoch AI—
Llama 2 13B39.4%Self-reportedLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Llama 2 7B39.2%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Baichuan2-13B-Chat38.8%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
MPT-30B (Instruct, Chat)38.0%Self-reportedQwen Technical Report via Epoch AI—
Falcon-40B Instruct37.1%Self-reportedQwen Technical Report via Epoch AI—
InternLM-Chat-20B36.7%Third partyYi: Open Foundation Models by 01.AI via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)35.6%Self-reportedQwen Technical Report via Epoch AI—
Gemma 2B35.2%Self-reportedGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)35.2%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
ChatGLM2-6B33.7%Self-reportedQwen Technical Report via Epoch AI—
ChatGLM2-6B33.7%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Llama 2 7B32.6%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Llama 2 7B32.6%Self-reportedLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Baichuan-7B (base)32.5%Self-reportedBaichuan 2: Open Large-scale Language Models via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)31.0%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Falcon-7B Instruct28.8%Third partyBaichuan 2: Open Large-scale Language Models via Epoch AI—
Falcon-7B Instruct28.0%Self-reportedQwen Technical Report via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources