supermarket

Benchmark

LAMBADA

Measures: Predict the last word of a narrative passage, where the word can only be guessed from the whole passage and not from the final sentence alone.

Maintainer
Paperno et al.
Introduced
2016
Status
Unmaintained
Saturation
Near-ceiling
Typical provenance
Self-reported
In scoring set
No

Paper

Leaderboard

Show independent results only

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

Leaderboard
ModelAccuracyProvenanceSourceEvaluated
text-davinci-001 (InstructGPT)
OpenAI
86.4%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
86.4%Third partyUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI—
Falcon-180B Chat
Technology Innovation Institute (TII)
79.8%Self-reportedThe Falcon Series of Open Language Models via Epoch AI—
Inflection-1
Inflection AI
78.5%Third partyThe Falcon Series of Open Language Models via Epoch AI—
Falcon-40B Instruct
Technology Innovation Institute (TII)
77.3%Self-reportedThe Falcon Series of Open Language Models via Epoch AI—
Llama 2 13B
Meta
76.5%Self-reportedQwen Technical Report via Epoch AI—
Llama 2 13B
Meta
76.5%Self-reportedQwen Technical Report via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
76.2%Third partyUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
76.2%Third partyThe Falcon Series of Open Language Models via Epoch AI—
Falcon-7B Instruct
Technology Innovation Institute (TII)
74.9%Self-reportedThe Falcon Series of Open Language Models via Epoch AI—
Llama 2 7B
Meta
73.3%Self-reportedQwen Technical Report via Epoch AI—
Llama 2 7B
Meta
73.3%Self-reportedQwen Technical Report via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
72.5%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
72.5%Third partyUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI—
Stable Beluga 2 (Llama 2 70B fine-tune)
Stability AI
71.3%Self-reportedQwen Technical Report via Epoch AI—
Stable Beluga 2 (Llama 2 70B fine-tune)
Stability AI
71.3%Self-reportedQwen Technical Report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)
Databricks
70.0%Self-reportedQwen Technical Report via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)
Databricks
70.0%Self-reportedQwen Technical Report via Epoch AI—
ChatGLM2-6B
Z.ai (Zhipu AI)
54.3%Self-reportedQwen Technical Report via Epoch AI—
ChatGLM2-6B
Z.ai (Zhipu AI)
54.3%Self-reportedQwen Technical Report via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources