supermarket

Benchmark

WeirdML (v2)

Measures: Unusual machine-learning engineering tasks (shape recognition, digit classification, chess outcomes and others) where the model writes PyTorch code that is trained and scored automatically.

Maintainer
Håvard Tveit Ihle
Introduced
2025
Status
Active
Saturation
Saturated
Typical provenance
Independent
In scoring set
No

Official leaderboard

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
ModelAverage accuracyProvenanceSourceEvaluated
GPT-6 Astra
OpenAI · Promax effort
93.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-6 Astra
OpenAI · Max effort
93.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Fable 5.1
Anthropic · Max effort
92.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-6 Astra
OpenAI · High effort
92.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Fable 5.1
Anthropic · High effort
92.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Fable 5
Anthropic · Max effort
91.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 5
Anthropic · Max effort
91.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 5
Anthropic · High effort
91.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.6 Sol
OpenAI · Promax effort
89.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.6 Sol
OpenAI · High effort
88.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Fable 5
Anthropic · High effort
87.8%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5.6 Sol
OpenAI · Max effort
87.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 5
Anthropic
86.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.5
OpenAI · Xhigh effort
84.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.5
OpenAI · High effort
83.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.8
Anthropic · Xhigh effort
82.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Kimi K3
Moonshot AI · Max effort
82.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.3-Codex
OpenAI
79.3%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5.6 Terra
OpenAI · High effort
78.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.6
Anthropic · High effort
78.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.6
Anthropic · Unknown effort
77.9%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5.3-Codex
OpenAI · Xhigh effort
77.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.4
OpenAI · Xhigh effort
77.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.7
Anthropic · High effort
76.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.7
Anthropic
76.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.7
Anthropic · Unknown effort
76.4%Third partyWeirdML Leaderboard via Epoch AI—
Claude Opus 4.8
Anthropic · Medium effort
76.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.7
Anthropic · Max effort
75.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GLM-5.3
Z.ai (Zhipu AI) · Max effort
75.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.2
OpenAI · Xhigh effort
72.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3.1 Pro
Google DeepMind
72.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.8
Anthropic · None effort
70.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GLM-5.2
Z.ai (Zhipu AI) · Max effort
70.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3 Pro
Google DeepMind
69.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Sonnet 5
Anthropic · High effort
68.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GLM-5.2
Z.ai (Zhipu AI) · High effort
67.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 4.6
SpaceXAI (xAI) · High effort
67.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.5
OpenAI · None effort
67.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Pro
DeepSeek · Max effort
66.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Sonnet 4.6
Anthropic · Medium effort
66.1%Third partyWeirdML Leaderboard via Epoch AI—
Claude Opus 4.6
Anthropic
65.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.5
Anthropic · 16K effort
63.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.2
OpenAI · Medium effort
63.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Flash
DeepSeek · Max effort
63.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3.5 Flash
Google DeepMind · High effort
62.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3 Flash
Google DeepMind
61.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.6 Luna
OpenAI · High effort
60.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.1
OpenAI · High effort
60.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5
OpenAI · High effort
60.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5 Pro
OpenAI · High effort
60.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.4 mini
OpenAI · High effort
60.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Muse Spark 1.2
Meta · Xhigh effort
60.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o3-pro
OpenAI · High effort
58.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.4
OpenAI · None effort
57.4%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5.4 Pro
OpenAI · None effort
57.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GLM-5.1
Z.ai (Zhipu AI)
57.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Flash
DeepSeek · High effort
57.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3.6 Flash
Google DeepMind · High effort
56.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Kimi K2.6
Moonshot AI
55.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5-Codex
OpenAI
54.5%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5-Codex
OpenAI · High effort
54.5%Third partyWeirdML Leaderboard via Epoch AI—
Kimi K2.7 Code
Moonshot AI
54.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 2.5 Pro
Google DeepMind · 16K effort
54.0%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5 mini
OpenAI · High effort
52.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o4-mini
OpenAI · High effort
52.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o3
OpenAI · High effort
52.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemma 4 31B
Google DeepMind
52.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 4.20
SpaceXAI (xAI)
52.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3.1 Flash-Lite
Google DeepMind
52.2%Third partyWeirdML Leaderboard via Epoch AI—
Grok 4.3
SpaceXAI (xAI) · Unknown effort
49.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.2
OpenAI · None effort
49.6%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5.2
OpenAI · Low effort
49.6%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5.4 nano
OpenAI · High effort
49.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Pro
DeepSeek · Max effort
48.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GLM-5
Z.ai (Zhipu AI)
48.2%Third partyWeirdML Leaderboard via Epoch AI—
gpt-oss-120b
OpenAI · High effort
48.2%Third partyWeirdML Leaderboard via Epoch AI—
gpt-oss-120b
OpenAI · High effort
48.2%Third partyWeirdML Leaderboard via Epoch AI—
Claude Sonnet 4.5
Anthropic · 16K effort
47.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o1-preview
OpenAI
47.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V3.2-Speciale
DeepSeek
46.7%Third partyWeirdML Leaderboard via Epoch AI—
Claude Sonnet 4.5
Anthropic
46.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Pro
DeepSeek · High effort
46.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 4.5
SpaceXAI (xAI) · Unknown effort
46.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Sonnet 4
Anthropic · 16K effort
46.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o1
OpenAI · High effort
46.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4.1
Anthropic · 16K effort
45.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 4
SpaceXAI (xAI)
45.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Flash
DeepSeek · Max effort
45.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Kimi K2.5
Moonshot AI
45.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Haiku 4.5
Anthropic
45.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Haiku 4.5
Anthropic · 16K effort
44.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Sonnet 4
Anthropic
43.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V4-Flash
DeepSeek · High effort
43.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude Opus 4
Anthropic · 16K effort
43.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Mistral Medium 3.5
Mistral AI
43.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o3-mini
OpenAI · High effort
43.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Mercury 2
Inception
43.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 4 Fast
SpaceXAI (xAI)
42.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Kimi K2 Thinking
Moonshot AI
42.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 3 Mini
SpaceXAI (xAI) · High effort
42.6%Third partyWeirdML Leaderboard via Epoch AI—
Grok 3 Mini
SpaceXAI (xAI) · High effort
42.6%Third partyWeirdML Leaderboard via Epoch AI—
Gemini 2.5 Flash
Google DeepMind · 16k effort
41.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
gpt-oss-120b
OpenAI · Medium effort
41.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-R1
DeepSeek
41.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3-Coder-480B-A35B-Instruct
Alibaba Group
41.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3-235B-A22B-Thinking-2507
Alibaba Group
41.0%Third partyWeirdML Leaderboard via Epoch AI—
Gemini 2.5 Flash
Google DeepMind · 16K effort
41.0%Third partyWeirdML Leaderboard via Epoch AI—
gpt-oss-20b
OpenAI · High effort
40.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GLM-4.5
Z.ai (Zhipu AI) · Thinking effort
40.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 3.5 Sonnet
Anthropic
40.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5 Instant (gpt-5-chat-latest)
OpenAI
39.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3.5-27B
Alibaba Group
39.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V3.2-Exp
DeepSeek · Thinking effort
39.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4.5
OpenAI
39.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Kimi K2
Moonshot AI
39.4%Third partyWeirdML Leaderboard via Epoch AI—
GPT-4.1
OpenAI
39.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 3.5 Flash-Lite
Google DeepMind · High effort
39.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3-235B-A22B-Instruct-2507
Alibaba Group
38.7%Third partyWeirdML Leaderboard via Epoch AI—
Qwen3-235B-A22B-Thinking-2507
Alibaba Group
38.7%Third partyWeirdML Leaderboard via Epoch AI—
DeepSeek-V3.1
DeepSeek · Thinking effort
38.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5 nano
OpenAI · High effort
38.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.4 nano
OpenAI · None effort
38.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-5.4 mini
OpenAI · None effort
37.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4.1 mini
OpenAI
37.6%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-V3.1
DeepSeek
37.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3-235B-A22B
Alibaba Group
37.3%Third partyWeirdML Leaderboard via Epoch AI—
Qwen3-235B-A22B-Thinking-2507
Alibaba Group
37.3%Third partyWeirdML Leaderboard via Epoch AI—
Grok 3
SpaceXAI (xAI)
37.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
MiniMax-M2.7
MiniMax
37.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
gpt-oss-20b
OpenAI · Medium effort
36.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Kimi K2
Moonshot AI
36.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
DeepSeek-R1
DeepSeek
36.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
o1-mini
OpenAI · Medium effort
36.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 2.5 Flash-Lite
Google DeepMind · 16K effort
35.2%Third partyWeirdML Leaderboard via Epoch AI—
Gemma 4 26B A4B
Google DeepMind
35.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok Code Fast 1
SpaceXAI (xAI)
35.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3.6-35B-A3B
Alibaba Group
34.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3-Coder-Next
Alibaba Group
34.4%Third partyWeirdML Leaderboard via Epoch AI—
Mistral Medium 3.1
Mistral AI
33.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Inkling
Thinking Machines Lab · High effort
32.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 3.5 Sonnet
Anthropic
31.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 3.5 Haiku
Anthropic
30.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen3-30B-A3B
Alibaba Group
29.8%Third partyWeirdML Leaderboard via Epoch AI—
GPT-5 nano
OpenAI · Low effort
25.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 2.0 Flash
Google DeepMind
25.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4o
OpenAI
25.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 1.5 Flash
Google DeepMind
24.9%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Llama 4 Maverick
Meta
24.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Grok 2
SpaceXAI (xAI)
22.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Gemini 1.5 Pro
Google DeepMind
22.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Llama 3.1 405B
Meta
21.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 3 Opus
Anthropic
19.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4.1 nano
OpenAI
19.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4 Turbo
OpenAI
18.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen2.5-72B-Instruct
Alibaba Group
16.0%Third partyWeirdML Leaderboard via Epoch AI—
Llama 3.3 70B
Meta
14.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4
OpenAI
12.4%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-4o mini
OpenAI
11.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Qwen2-72B-Instruct
Alibaba Group
11.3%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 3 Sonnet
Anthropic
10.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 3 Haiku
Anthropic
9.8%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Llama 3.1 70B
Meta
9.0%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Claude 2.1
Anthropic
7.1%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
GPT-3.5 Turbo
OpenAI
3.5%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Mixtral 8x22B
Mistral AI
3.2%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
Llama 3.1 8B
Meta
1.7%Third partyhttps://htihle.github.io/weirdml.html via Epoch AI—
  • Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

WeirdML (v2): leaderboard and what it measures · supermarket.dev