supermarket

Vision-language

Gemini 1.5 Flash

By Google DeepMind, in the Gemini Flash family.

Released
14 May 2024
Status
Retired
Retired
29 Sep 2025
Context
1M
Max output
8.2K
Input per 1M tokens
$0.07
Output per 1M tokens
$0.30
Open weights
No
License
proprietary
Parameters
—
Knowledge cutoff
Nov 2023
Reasoning
No
Modalities
text, image, audio, video → text

Category scores

Out of 100, from independent and third-party results only, by method category-2026-09-29.1.

Category scores
CategoryScoreRankCoverage
Agentic tool use14.6Not ranked1 of 2
Vision & multimodal understanding76.0Not ranked1 of 3
Human preference27.8Not ranked2 of 5

Benchmark results

Every result we hold for this model, with who ran it and where it was published.

Knowledge & reasoning

Knowledge & reasoning results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
DTBench (Decision Theory Benchmark)Accuracy (capability questions)53.8%Third party——https://conceptualreasoning.ai/dtbench via Epoch AI
DTBench (Decision Theory Benchmark)Accuracy (capability questions)51.4%Third party——https://conceptualreasoning.ai/dtbench via Epoch AI
MMLUAccuracy77.9%Third party——Stanford CRFM Leaderboard via Epoch AI
MMLUAccuracy73.9%Third party——Stanford CRFM Leaderboard via Epoch AI
PIQAAccuracy87.5%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI

Scientific reasoning

Scientific reasoning results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
GPQA DiamondAccuracy47.3%Independent—27 Jan 2025Epoch AI
GPQA DiamondAccuracy40.4%Independent—27 Jan 2025Epoch AI

Mathematics

Mathematics results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
FrontierMath Tiers 1-3 (v1)Accuracy0.0%Independent—7 Mar 2025Epoch AI
GSM8KExact match82.4%Self-reported——Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone via Epoch AI
MATH Level 5Accuracy61.9%Independent—27 Jan 2025Epoch AI
MATH Level 5Accuracy25.1%Independent—27 Jan 2025Epoch AI
OTIS Mock AIME 2024-2025Accuracy16.3%Independent—25 Feb 2025Epoch AI
OTIS Mock AIME 2024-2025Accuracy3.9%Independent—25 Feb 2025Epoch AI

Coding & software engineering

Coding & software engineering results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
WeirdML (v2)Average accuracy24.9%Third party——https://htihle.github.io/weirdml.html via Epoch AI

Agentic tool use

Agentic tool use results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
BALROGAverage progress14.6%Third party——https://balrogai.com/ via Epoch AI

Vision & multimodal understanding

Vision & multimodal understanding results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
GeoBenchCountry accuracy76.0%Third party——GeoBench leaderboard via Epoch AI

Human preference

Human preference results
BenchmarkMetricValueProvenanceSettingEvaluatedSource
Text Arena (arena.ai)Arena score (Elo scale)1309Independent—25 Sep 2026Arena
Text Arena (arena.ai)Arena score (Elo scale)1287Independent—25 Sep 2026Arena
Text Arena (arena.ai)Votes62833Independent—25 Sep 2026Arena
Text Arena (arena.ai)Votes34902Independent—25 Sep 2026Arena
Vision Arena (arena.ai)Arena score (Elo scale)1141Independent—28 Sep 2026Arena
Vision Arena (arena.ai)Arena score (Elo scale)1060Independent—28 Sep 2026Arena
Vision Arena (arena.ai)Votes13260Independent—28 Sep 2026Arena
Vision Arena (arena.ai)Votes7241Independent—28 Sep 2026Arena

Other Gemini Flash versions

Sources