Benchmark

HellaSwag

Measures: Commonsense sentence completion: pick the most plausible of four endings for an everyday scene, with wrong endings adversarially filtered to fool models.

Maintainer
Zellers et al. (University of Washington and AI2)
Introduced
2019
Status
Unmaintained
Saturation
Saturated
Typical provenance
Self-reported
In scoring set
No

Paper

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

Leaderboard
ModelAccuracyProvenanceSourceEvaluated
GPT-495.3%Self-reportedGPT-4 Technical Report via Epoch AI—
GPT-4 32K95.3%Third partyThe Falcon Series of Open Language Models via Epoch AI—
Llama 3.1 405B89.2%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
Falcon-180B Chat89.0%Self-reportedThe Falcon Series of Open Language Models via Epoch AI—
DeepSeek-V388.9%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
DeepSeek-V287.1%Self-reportedDeepSeek-V3 Technical Report via Epoch AI—
text-davinci-00385.5%Third partyThe Falcon Series of Open Language Models via Epoch AI—
text-davinci-00385.5%Self-reportedGPT-4 Technical Report via Epoch AI—
Falcon-40B Instruct85.3%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon-40B Instruct85.3%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Stable Beluga 2 (Llama 2 70B fine-tune)84.1%Self-reportedQwen Technical Report via Epoch AI—
Falcon 2 11B82.9%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon 2 11B82.9%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon-40B Instruct82.8%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon-40B Instruct82.8%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Phi-3-medium82.4%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Gemma 7B82.2%Self-reportedGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
text-davinci-00382.2%Third partyStanford HELM via Epoch AI—
Falcon 2 11B82.1%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon 2 11B82.1%Self-reportedFalcon2-11B Technical Report via Epoch AI—
text-davinci-00281.5%Third partyStanford HELM via Epoch AI—
Gemma 7B81.2%Self-reportedGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Mistral 7B v0.181.0%Third partyMixtral of Experts via Epoch AI—
Llama 2 13B80.7%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
text-davinci-001 (InstructGPT)79.3%Third partyUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI—
text-davinci-001 (InstructGPT)79.3%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)79.3%Third partyLanguage Models are Few-Shot Learners via Epoch AI—
text-davinci-001 (InstructGPT)78.9%Third partyUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI—
text-davinci-001 (InstructGPT)78.9%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)78.9%Third partyLanguage Models are Few-Shot Learners via Epoch AI—
Falcon-7B Instruct78.1%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon-7B Instruct78.1%Self-reportedFalcon2-11B Technical Report via Epoch AI—
text-davinci-001 (InstructGPT)78.1%Third partyUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI—
text-davinci-001 (InstructGPT)78.1%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)78.1%Third partyLanguage Models are Few-Shot Learners via Epoch AI—
GPT-3 davinci77.5%Third partyStanford HELM via Epoch AI—
Llama 2 7B77.2%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Falcon-7B Instruct76.4%Self-reportedXGen-7B Technical Report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)76.4%Self-reportedQwen Technical Report via Epoch AI—
Yi-9B76.4%Self-reportedYi: Open Foundation Models by 01.AI via Epoch AI—
Falcon-7B Instruct76.3%Self-reportedFalcon2-11B Technical Report via Epoch AI—
Falcon-7B Instruct76.3%Self-reportedFalcon2-11B Technical Report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)76.1%Self-reportedXGen-7B Technical Report via Epoch AI—
OPT-66B74.5%Third partyStanford HELM via Epoch AI—
Falcon-7B Instruct74.1%Self-reportedQwen Technical Report via Epoch AI—
Gemma 2B71.4%Self-reportedGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Dolly 2.0 (3B, 7B, 12B)70.8%Self-reportedXGen-7B Technical Report via Epoch AI—
OPT-13B69.9%Self-reportedXGen-7B Technical Report via Epoch AI—
GPT-3 curie68.2%Third partyStanford HELM via Epoch AI—
text-curie-001 (InstructGPT)67.6%Third partyStanford HELM via Epoch AI—
Llama 2 7B57.1%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)57.1%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
ChatGLM2-6B57.0%Self-reportedQwen Technical Report via Epoch AI—
text-babbage-001 (InstructGPT)56.1%Third partyStanford HELM via Epoch AI—
GPT-3 babbage55.5%Third partyStanford HELM via Epoch AI—
Falcon-7B Instruct54.2%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
Phi-253.6%Self-reportedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI—
Phi-1.5
Microsoft · 5 effort
47.6%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
GPT-3 ada43.5%Third partyStanford HELM via Epoch AI—
text-ada-001 (InstructGPT)42.9%Third partyStanford HELM via Epoch AI—
OPT-1.3B41.5%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
StableLM-Alpha 7B40.7%Self-reportedXGen-7B Technical Report via Epoch AI—
GPT-2 XL (1.5B)40.0%Self-reportedTextbooks Are All You Need II: phi-1.5 technical report via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources