supermarket

Vision-language

Gemma 7B

By Google DeepMind, in the Gemma family.

Released
21 Feb 2024
Status
Legacy
Context
8.2K
Max output
—
Input per 1M tokens
—
Output per 1M tokens
—
Open weights
Yes
License
gemma
Parameters
8.54B
Knowledge cutoff
—
Reasoning
No
Modalities
text → text

Category scores

Out of 100, from independent and third-party results only, by method category-2026-09-29.1.

Category scores
CategoryScoreRankCoverage
Human preference18.1Not ranked1 of 5

Benchmark results

Every result we hold for this model, with who ran it and where it was published.

Knowledge & reasoning

Knowledge & reasoning results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Adversarial NLI (ANLI)Accuracy48.7%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI
AI2 Reasoning Challenge (ARC)Challenge set accuracy78.3%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI
AI2 Reasoning Challenge (ARC)Challenge set accuracy53.2%Self-reported——Nemotron-4 15B Technical Report via Epoch AI
AI2 Reasoning Challenge (ARC)Challenge set accuracy53.2%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
BIG-Bench Hard (BBH)Average accuracy55.1%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
BIG-Bench Hard (BBH)Average accuracy55.1%Self-reported——Nemotron-4 15B Technical Report via Epoch AI
HellaSwagAccuracy82.2%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
HellaSwagAccuracy81.2%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
MMLUAccuracy66.1%Third party——Stanford CRFM Leaderboard via Epoch AI
MMLUAccuracy64.3%Self-reported——Nemotron-4 15B Technical Report via Epoch AI
MMLUAccuracy64.3%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
MMLUAccuracy63.6%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI
OpenBookQAAccuracy78.6%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI
PIQAAccuracy81.2%Self-reported——Nemotron-4 15B Technical Report via Epoch AI
PIQAAccuracy81.2%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
WinoGrandeAccuracy79.0%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI
WinoGrandeAccuracy72.3%Self-reported——Nemotron-4 15B Technical Report via Epoch AI
WinoGrandeAccuracy55.6%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI

Mathematics

Mathematics results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
GSM8KExact match46.4%Self-reported——Nemotron-4 15B Technical Report via Epoch AI
GSM8KExact match46.4%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI

Factuality & hallucination

Factuality & hallucination results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
TriviaQAExact match72.3%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI
TriviaQAExact match63.4%Self-reported——Gemma: Open Models Based on Gemini Research and Technology via Epoch AI

Human preference

Human preference results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Text Arena (arena.ai)Arena score (Elo scale)1138Independent—25 Sep 2026Arena
Text Arena (arena.ai)Votes8925Independent—25 Sep 2026Arena

Other Gemma versions

Sources