Benchmark

TheAgentCompany

Measures: Tasks from a simulated software company (engineering, project management, HR, finance) that an agent does by browsing, coding and messaging simulated colleagues; scored on tasks fully resolved.

Maintainer
TheAgentCompany team (Carnegie Mellon University)
Introduced
2024
Status
Active
Saturation
Headroom
Typical provenance
Independent
In scoring set
Yes

PaperOfficial leaderboard

Leaderboard

Independent and third-party results only.

Leaderboard
ModelTasks resolvedProvenanceSourceEvaluated
DeepSeek-V3.2-Exp42.9%Third partyTheAgentCompany leaderboard via Epoch AI—
Gemini 2.5 Flash41.1%Third partyTheAgentCompany leaderboard via Epoch AI—
Claude Sonnet 433.1%Third partyTheAgentCompany leaderboard via Epoch AI—
Claude 3.7 Sonnet30.9%Third partyTheAgentCompany leaderboard via Epoch AI—
Gemini 2.5 Pro30.3%Third partyTheAgentCompany leaderboard via Epoch AI—
Claude 3.7 Sonnet26.3%Third partyTheAgentCompany leaderboard via Epoch AI—
Claude 3.5 Sonnet24.0%Third partyTheAgentCompany experiment results github via Epoch AI—
Gemini 2.0 Flash11.4%Third partyTheAgentCompany experiment results github via Epoch AI—
GPT-4o8.6%Third partyTheAgentCompany experiment results github via Epoch AI—
Llama 3.1 405B7.4%Third partyTheAgentCompany experiment results github via Epoch AI—
Llama 3.1 70B6.9%Third partyTheAgentCompany experiment results github via Epoch AI—
Qwen2.5-72B-Instruct5.7%Third partyTheAgentCompany experiment results github via Epoch AI—
Gemini 1.5 Pro3.4%Third partyTheAgentCompany experiment results github via Epoch AI—
Amazon Nova Pro1.7%Third partyTheAgentCompany experiment results github via Epoch AI—
Llama 3.1 70B1.7%Third partyTheAgentCompany experiment results github via Epoch AI—
Qwen2-72B-Instruct1.1%Third partyTheAgentCompany experiment results github via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

TheAgentCompany: leaderboard and what it measures · supermarket.dev