SWE-bench Pro

Measures: An agent must produce a patch for a long-horizon bug fix or feature request (about 107 changed lines over 4 files on average) in copyleft-licensed or private startup repositories in Python, Go, JavaScript and TypeScript; scored by resolve rate, the percentage of tasks whose fail-to-pass and pass-to-pass tests all pass.

Maintainer: Scale AI (V2 co-developed with Reflection)
Introduced: 2025
Status: Active
Saturation: Saturated
Typical provenance: Mixed
In scoring set: No

Paper Official leaderboard

Leaderboard

Show independent results only

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

No results to show.