MultiChallenge

Measures: Realistic multi-turn conversations test four abilities: keeping instructions from the start of the chat, recalling scattered details, handling versioned edits reliably, and staying self-coherent without sycophancy. An LLM judge (Gemini 2.5 Pro since the Feb 2026 update) scores each conversation against human-written, instance-level yes/no rubric questions.

Maintainer: Scale AI (Scale Labs / SEAL)
Introduced: 2025
Status: Active
Saturation: Headroom
Typical provenance: Independent
In scoring set: Yes

Paper Official leaderboard

Leaderboard

Show independent results only

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

No results to show.