Updated daily at 05:00 UTC

Benchmark

BIG-Bench Hard (BBH)

Measures: 23 BIG-bench tasks on which language models had trailed average human raters, probing multi-step symbolic, logical and arithmetic reasoning.

Maintainer: Suzgun et al. (Google Research and Stanford)
Introduced: 2022
Status: Unmaintained
Saturation: Near-ceiling
Typical provenance: Self-reported
In scoring set: No

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
Model	Average accuracy	Provenance	Source	Evaluated
Gemini 1.5 Pro Google DeepMind	89.2%	Third party	Gemini 1.5 report via Epoch AI	—
GPT-4 OpenAI	75.1%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
GPT-3.5 Turbo OpenAI	61.6%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
Mistral 7B v0.1 Mistral AI	56.1%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
Qwen-14B-Chat Alibaba Group	55.0%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Qwen-14B-Chat Alibaba Group	49.7%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Baichuan2-13B-Chat Baichuan Intelligence	47.2%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Llama 2 13B Meta	47.0%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
InternLM-Chat-20B Shanghai AI Laboratory	42.4%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
Llama 2 13B Meta	39.4%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
Llama 2 7B Meta	39.2%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
Baichuan2-13B-Chat Baichuan Intelligence	38.8%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
InternLM-Chat-20B Shanghai AI Laboratory	36.7%	Third party	Yi: Open Foundation Models by 01.AI via Epoch AI	—
MPT-7B (Instruct, Chat, StoryWriter) Databricks	35.2%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
ChatGLM2-6B Z.ai (Zhipu AI)	33.7%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—
Llama 2 7B Meta	32.6%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
MPT-7B (Instruct, Chat, StoryWriter) Databricks	31.0%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Falcon-7B Instruct Technology Innovation Institute (TII)	28.8%	Third party	Baichuan 2: Open Large-scale Language Models via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI