WinoGrande

Measures: 44,000 Winograd-style fill-in-the-blank sentences with two options, built at scale and filtered to reduce annotation shortcuts.

Maintainer: Sakaguchi et al. (AI2 and University of Washington)
Introduced: 2019
Status: Unmaintained
Saturation: Headroom
Typical provenance: Self-reported
In scoring set: No

Paper

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
Model	Accuracy	Provenance	Source	Evaluated
Falcon-7B Instruct Technology Innovation Institute (TII)	66.3%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
MPT-7B (Instruct, Chat, StoryWriter) Databricks	68.3%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Llama 2 7B Meta	69.2%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	70.2%	Third party	LLaMA: Open and Efficient Foundation Language Models via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	70.2%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	70.2%	Third party	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	70.2%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	70.2%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
MPT-30B (Instruct, Chat) Databricks	71.0%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
Llama 2 13B Meta	72.8%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	73.2%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	73.2%	Third party	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	73.2%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	73.2%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
Mistral 7B v0.1 Mistral AI	74.2%	Third party	Mixtral of Experts via Epoch AI	—
Mistral 7B v0.1 Mistral AI	74.2%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
Mistral 7B v0.1 Mistral AI	75.3%	Third party	Training Compute-Optimal Large Language Models via Epoch AI	—
Falcon-40B Instruct Technology Innovation Institute (TII)	76.9%	Third party	Llama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	77.7%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	77.7%	Third party	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	77.7%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	77.7%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
GPT-3.5 Turbo OpenAI	81.6%	Third party	The Falcon Series of Open Language Models via Epoch AI	—
GPT-4 32K OpenAI	87.5%	Third party	The Falcon Series of Open Language Models via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI