OpenAI MRCR v2 (8-needle)

Measures: The model reads a long synthetic multi-turn conversation that hides 8 near-identical requests and must reproduce the i-th matching response, prefixed by a required hash; scored by difflib SequenceMatcher ratio and reported per context-length bin from 4K up to 1M tokens.

Maintainer: OpenAI (openai/mrcr dataset); the MRCR task comes from Google DeepMind's Michelangelo paper, and Google DeepMind maintains its own separate MRCR v2 in eval_hub
Introduced: 2025
Status: Active
Saturation: Near-ceiling
Typical provenance: Self-reported
In scoring set: No

Paper

Leaderboard

Show independent results only

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

No results to show.