Adversarial NLI (ANLI)

Measures: Natural language inference examples written adversarially against strong models over three rounds; the model says whether a hypothesis follows from, contradicts or is neutral to a premise.

Maintainer: Nie et al. (Facebook AI and UNC)
Introduced: 2019
Status: Unmaintained
Saturation: Near-ceiling
Typical provenance: Self-reported
In scoring set: No

Paper

Leaderboard

Show independent results only

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

Leaderboard
Model	Accuracy	Provenance	Source	Evaluated
text-davinci-001 (InstructGPT) OpenAI	33.9%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	34.0%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
text-davinci-001 (InstructGPT) OpenAI	35.4%	Third party	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model via Epoch AI	—
Phi-2 Microsoft	42.5%	Self-reported	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI	—
Mistral 7B v0.1 Mistral AI	47.1%	Self-reported	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI	—
Gemma 7B Google DeepMind	48.7%	Self-reported	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI	—
Phi-3-medium Microsoft	55.8%	Self-reported	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI	—
Llama 3 8B Meta	57.3%	Self-reported	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI	—
GPT-3.5 Turbo OpenAI	58.1%	Self-reported	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI