HellaSwag

Measures: Commonsense sentence completion: pick the most plausible of four endings for an everyday scene, with wrong endings adversarially filtered to fool models.

Maintainer: Zellers et al. (University of Washington and AI2)
Introduced: 2019
Status: Unmaintained
Saturation: Saturated
Typical provenance: Self-reported
In scoring set: No

Paper

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
Model	Accuracy	Provenance	Source	Evaluated
GPT-4 32K OpenAI	95.3%	Third party	The Falcon Series of Open Language Models via Epoch AI	—
text-davinci-003 OpenAI	85.5%	Third party	The Falcon Series of Open Language Models via Epoch AI	—
text-davinci-003 OpenAI	82.2%	Third party	Stanford HELM via Epoch AI	—
text-davinci-002 OpenAI	81.5%	Third party	Stanford HELM via Epoch AI	—
Mistral 7B v0.1 Mistral AI	81.0%	Third party	Mixtral of Experts via Epoch AI	—
Llama 2 13B Meta	80.7%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	79.3%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	79.3%	Third party	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	79.3%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	78.9%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	78.9%	Third party	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	78.9%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	78.1%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	78.1%	Third party	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	78.1%	Third party	Language Models are Few-Shot Learners via Epoch AI	—
GPT-3 davinci OpenAI	77.5%	Third party	Stanford HELM via Epoch AI	—
Llama 2 7B Meta	77.2%	Third party	Gemma: Open Models Based on Gemini Research and Technology via Epoch AI	—
OPT-66B Meta	74.5%	Third party	Stanford HELM via Epoch AI	—
GPT-3 curie OpenAI	68.2%	Third party	Stanford HELM via Epoch AI	—
text-curie-001 (InstructGPT) OpenAI	67.6%	Third party	Stanford HELM via Epoch AI	—
text-babbage-001 (InstructGPT) OpenAI	56.1%	Third party	Stanford HELM via Epoch AI	—
GPT-3 babbage OpenAI	55.5%	Third party	Stanford HELM via Epoch AI	—
GPT-3 ada OpenAI	43.5%	Third party	Stanford HELM via Epoch AI	—
text-ada-001 (InstructGPT) OpenAI	42.9%	Third party	Stanford HELM via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI