Benchmark

TheAgentCompany

Measures: Tasks from a simulated software company (engineering, project management, HR, finance) that an agent does by browsing, coding and messaging simulated colleagues; scored on tasks fully resolved.

Maintainer
TheAgentCompany team (Carnegie Mellon University)
Introduced
2024
Status
Active
Saturation
Headroom
Typical provenance
Independent
In scoring set
Yes

PaperOfficial leaderboard

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

Leaderboard
ModelTasks resolvedProvenanceSourceEvaluated
Qwen2-72B-Instruct1.1%Third partyTheAgentCompany experiment results github via Epoch AI—
Amazon Nova Pro1.7%Third partyTheAgentCompany experiment results github via Epoch AI—
Llama 3.1 70B1.7%Third partyTheAgentCompany experiment results github via Epoch AI—
Gemini 1.5 Pro3.4%Third partyTheAgentCompany experiment results github via Epoch AI—
Qwen2.5-72B-Instruct5.7%Third partyTheAgentCompany experiment results github via Epoch AI—
Llama 3.1 70B6.9%Third partyTheAgentCompany experiment results github via Epoch AI—
Llama 3.1 405B7.4%Third partyTheAgentCompany experiment results github via Epoch AI—
GPT-4o8.6%Third partyTheAgentCompany experiment results github via Epoch AI—
Gemini 2.0 Flash11.4%Third partyTheAgentCompany experiment results github via Epoch AI—
Claude 3.5 Sonnet24.0%Third partyTheAgentCompany experiment results github via Epoch AI—
Claude 3.7 Sonnet26.3%Third partyTheAgentCompany leaderboard via Epoch AI—
Gemini 2.5 Pro30.3%Third partyTheAgentCompany leaderboard via Epoch AI—
Claude 3.7 Sonnet30.9%Third partyTheAgentCompany leaderboard via Epoch AI—
Claude Sonnet 433.1%Third partyTheAgentCompany leaderboard via Epoch AI—
Gemini 2.5 Flash41.1%Third partyTheAgentCompany leaderboard via Epoch AI—
DeepSeek-V3.2-Exp42.9%Third partyTheAgentCompany leaderboard via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

TheAgentCompany: leaderboard and what it measures · supermarket.dev