supermarket

Benchmark

AI2 Reasoning Challenge (ARC)

Measures: Grade-school science multiple-choice questions (four options); the Challenge set keeps the questions that simple retrieval and word co-occurrence methods got wrong.

Maintainer
Allen Institute for AI
Introduced
2018
Status
Unmaintained
Saturation
Saturated
Typical provenance
Self-reported
In scoring set
No

Paper

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
ModelChallenge set accuracyProvenanceSourceEvaluated
Falcon-7B Instruct
Technology Innovation Institute (TII)
42.4%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
MPT-7B (Instruct, Chat, StoryWriter)
Databricks
42.6%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Llama 2 7B
Meta
43.2%Third partyMixtral of Experts via Epoch AI—
Llama 2 7B
Meta
45.9%Third partyThe Falcon Series of Open Language Models via Epoch AI—
Llama 2 7B
Meta
45.9%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
Llama 2 13B
Meta
48.8%Third partyMixtral of Experts via Epoch AI—
Llama 2 13B
Meta
49.4%Third partyThe Falcon Series of Open Language Models via Epoch AI—
Llama 2 13B
Meta
49.4%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
MPT-30B (Instruct, Chat)
Databricks
50.6%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
51.4%Third partyThe Falcon Series of Open Language Models via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
51.4%Third partyLLaMA: Open and Efficient Foundation Language Models via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
51.4%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
51.5%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
text-davinci-001 (InstructGPT)
OpenAI
53.2%Third partyGLaM: Efficient Scaling of Language Models with Mixture-of-Experts via Epoch AI—
Falcon-40B Instruct
Technology Innovation Institute (TII)
54.5%Third partyLlama 2: Open Foundation and Fine-Tuned Chat Models via Epoch AI—
Mistral 7B v0.1
Mistral AI
54.9%Third partyMixtral of Experts via Epoch AI—
Mistral 7B v0.1
Mistral AI
54.9%Third partyGemma: Open Models Based on Gemini Research and Technology via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

AI2 Reasoning Challenge (ARC): leaderboard and what it measures · supermarket.dev