WeirdML (v2)

Measures: Unusual machine-learning engineering tasks (shape recognition, digit classification, chess outcomes and others) where the model writes PyTorch code that is trained and scored automatically.

Maintainer: Håvard Tveit Ihle
Introduced: 2025
Status: Active
Saturation: Saturated
Typical provenance: Independent
In scoring set: No

Official leaderboard

Leaderboard

Show all results

Independent and third-party results only.

Leaderboard
Model	Average accuracy	Provenance	Source	Evaluated
GPT-6 Astra OpenAI · Promax effort	93.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-6 Astra OpenAI · Max effort	93.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Fable 5.1 Anthropic · Max effort	92.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-6 Astra OpenAI · High effort	92.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Fable 5.1 Anthropic · High effort	92.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Fable 5 Anthropic · Max effort	91.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 5 Anthropic · Max effort	91.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 5 Anthropic · High effort	91.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.6 Sol OpenAI · Promax effort	89.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.6 Sol OpenAI · High effort	88.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Fable 5 Anthropic · High effort	87.8%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5.6 Sol OpenAI · Max effort	87.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 5 Anthropic	86.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.5 OpenAI · Xhigh effort	84.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.5 OpenAI · High effort	83.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.8 Anthropic · Xhigh effort	82.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Kimi K3 Moonshot AI · Max effort	82.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.3-Codex OpenAI	79.3%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5.6 Terra OpenAI · High effort	78.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.6 Anthropic · High effort	78.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.6 Anthropic · Unknown effort	77.9%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5.3-Codex OpenAI · Xhigh effort	77.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.4 OpenAI · Xhigh effort	77.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.7 Anthropic · High effort	76.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.7 Anthropic	76.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.7 Anthropic · Unknown effort	76.4%	Third party	WeirdML Leaderboard via Epoch AI	—
Claude Opus 4.8 Anthropic · Medium effort	76.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.7 Anthropic · Max effort	75.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GLM-5.3 Z.ai (Zhipu AI) · Max effort	75.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.2 OpenAI · Xhigh effort	72.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3.1 Pro Google DeepMind	72.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.8 Anthropic · None effort	70.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GLM-5.2 Z.ai (Zhipu AI) · Max effort	70.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3 Pro Google DeepMind	69.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Sonnet 5 Anthropic · High effort	68.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GLM-5.2 Z.ai (Zhipu AI) · High effort	67.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 4.6 SpaceXAI (xAI) · High effort	67.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.5 OpenAI · None effort	67.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Pro DeepSeek · Max effort	66.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Sonnet 4.6 Anthropic · Medium effort	66.1%	Third party	WeirdML Leaderboard via Epoch AI	—
Claude Opus 4.6 Anthropic	65.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.5 Anthropic · 16K effort	63.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.2 OpenAI · Medium effort	63.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Flash DeepSeek · Max effort	63.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3.5 Flash Google DeepMind · High effort	62.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3 Flash Google DeepMind	61.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.6 Luna OpenAI · High effort	60.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.1 OpenAI · High effort	60.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5 OpenAI · High effort	60.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5 Pro OpenAI · High effort	60.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.4 mini OpenAI · High effort	60.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Muse Spark 1.2 Meta · Xhigh effort	60.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o3-pro OpenAI · High effort	58.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.4 OpenAI · None effort	57.4%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5.4 Pro OpenAI · None effort	57.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GLM-5.1 Z.ai (Zhipu AI)	57.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Flash DeepSeek · High effort	57.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3.6 Flash Google DeepMind · High effort	56.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Kimi K2.6 Moonshot AI	55.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5-Codex OpenAI	54.5%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5-Codex OpenAI · High effort	54.5%	Third party	WeirdML Leaderboard via Epoch AI	—
Kimi K2.7 Code Moonshot AI	54.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 2.5 Pro Google DeepMind · 16K effort	54.0%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5 mini OpenAI · High effort	52.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o4-mini OpenAI · High effort	52.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o3 OpenAI · High effort	52.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemma 4 31B Google DeepMind	52.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 4.20 SpaceXAI (xAI)	52.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3.1 Flash-Lite Google DeepMind	52.2%	Third party	WeirdML Leaderboard via Epoch AI	—
Grok 4.3 SpaceXAI (xAI) · Unknown effort	49.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.2 OpenAI · None effort	49.6%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5.2 OpenAI · Low effort	49.6%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5.4 nano OpenAI · High effort	49.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Pro DeepSeek · Max effort	48.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GLM-5 Z.ai (Zhipu AI)	48.2%	Third party	WeirdML Leaderboard via Epoch AI	—
gpt-oss-120b OpenAI · High effort	48.2%	Third party	WeirdML Leaderboard via Epoch AI	—
gpt-oss-120b OpenAI · High effort	48.2%	Third party	WeirdML Leaderboard via Epoch AI	—
Claude Sonnet 4.5 Anthropic · 16K effort	47.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o1-preview OpenAI	47.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V3.2-Speciale DeepSeek	46.7%	Third party	WeirdML Leaderboard via Epoch AI	—
Claude Sonnet 4.5 Anthropic	46.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Pro DeepSeek · High effort	46.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 4.5 SpaceXAI (xAI) · Unknown effort	46.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Sonnet 4 Anthropic · 16K effort	46.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o1 OpenAI · High effort	46.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4.1 Anthropic · 16K effort	45.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 4 SpaceXAI (xAI)	45.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Flash DeepSeek · Max effort	45.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Kimi K2.5 Moonshot AI	45.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Haiku 4.5 Anthropic	45.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Haiku 4.5 Anthropic · 16K effort	44.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Sonnet 4 Anthropic	43.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V4-Flash DeepSeek · High effort	43.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude Opus 4 Anthropic · 16K effort	43.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Mistral Medium 3.5 Mistral AI	43.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o3-mini OpenAI · High effort	43.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Mercury 2 Inception	43.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 4 Fast SpaceXAI (xAI)	42.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Kimi K2 Thinking Moonshot AI	42.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 3 Mini SpaceXAI (xAI) · High effort	42.6%	Third party	WeirdML Leaderboard via Epoch AI	—
Grok 3 Mini SpaceXAI (xAI) · High effort	42.6%	Third party	WeirdML Leaderboard via Epoch AI	—
Gemini 2.5 Flash Google DeepMind · 16k effort	41.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
gpt-oss-120b OpenAI · Medium effort	41.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-R1 DeepSeek	41.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3-Coder-480B-A35B-Instruct Alibaba Group	41.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3-235B-A22B-Thinking-2507 Alibaba Group	41.0%	Third party	WeirdML Leaderboard via Epoch AI	—
Gemini 2.5 Flash Google DeepMind · 16K effort	41.0%	Third party	WeirdML Leaderboard via Epoch AI	—
gpt-oss-20b OpenAI · High effort	40.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GLM-4.5 Z.ai (Zhipu AI) · Thinking effort	40.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 3.5 Sonnet Anthropic	40.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5 Instant (gpt-5-chat-latest) OpenAI	39.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3.5-27B Alibaba Group	39.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V3.2-Exp DeepSeek · Thinking effort	39.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4.5 OpenAI	39.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Kimi K2 Moonshot AI	39.4%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-4.1 OpenAI	39.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 3.5 Flash-Lite Google DeepMind · High effort	39.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3-235B-A22B-Instruct-2507 Alibaba Group	38.7%	Third party	WeirdML Leaderboard via Epoch AI	—
Qwen3-235B-A22B-Thinking-2507 Alibaba Group	38.7%	Third party	WeirdML Leaderboard via Epoch AI	—
DeepSeek-V3.1 DeepSeek · Thinking effort	38.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5 nano OpenAI · High effort	38.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.4 nano OpenAI · None effort	38.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-5.4 mini OpenAI · None effort	37.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4.1 mini OpenAI	37.6%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-V3.1 DeepSeek	37.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3-235B-A22B Alibaba Group	37.3%	Third party	WeirdML Leaderboard via Epoch AI	—
Qwen3-235B-A22B-Thinking-2507 Alibaba Group	37.3%	Third party	WeirdML Leaderboard via Epoch AI	—
Grok 3 SpaceXAI (xAI)	37.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
MiniMax-M2.7 MiniMax	37.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
gpt-oss-20b OpenAI · Medium effort	36.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Kimi K2 Moonshot AI	36.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
DeepSeek-R1 DeepSeek	36.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
o1-mini OpenAI · Medium effort	36.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 2.5 Flash-Lite Google DeepMind · 16K effort	35.2%	Third party	WeirdML Leaderboard via Epoch AI	—
Gemma 4 26B A4B Google DeepMind	35.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok Code Fast 1 SpaceXAI (xAI)	35.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3.6-35B-A3B Alibaba Group	34.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3-Coder-Next Alibaba Group	34.4%	Third party	WeirdML Leaderboard via Epoch AI	—
Mistral Medium 3.1 Mistral AI	33.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Inkling Thinking Machines Lab · High effort	32.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 3.5 Sonnet Anthropic	31.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 3.5 Haiku Anthropic	30.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen3-30B-A3B Alibaba Group	29.8%	Third party	WeirdML Leaderboard via Epoch AI	—
GPT-5 nano OpenAI · Low effort	25.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 2.0 Flash Google DeepMind	25.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4o OpenAI	25.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 1.5 Flash Google DeepMind	24.9%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Llama 4 Maverick Meta	24.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Grok 2 SpaceXAI (xAI)	22.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Gemini 1.5 Pro Google DeepMind	22.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Llama 3.1 405B Meta	21.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 3 Opus Anthropic	19.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4.1 nano OpenAI	19.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4 Turbo OpenAI	18.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen2.5-72B-Instruct Alibaba Group	16.0%	Third party	WeirdML Leaderboard via Epoch AI	—
Llama 3.3 70B Meta	14.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4 OpenAI	12.4%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-4o mini OpenAI	11.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Qwen2-72B-Instruct Alibaba Group	11.3%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 3 Sonnet Anthropic	10.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 3 Haiku Anthropic	9.8%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Llama 3.1 70B Meta	9.0%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Claude 2.1 Anthropic	7.1%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
GPT-3.5 Turbo OpenAI	3.5%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Mixtral 8x22B Mistral AI	3.2%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—
Llama 3.1 8B Meta	1.7%	Third party	https://htihle.github.io/weirdml.html via Epoch AI	—

Epoch AI, 'Capabilities & benchmarking', epoch.ai/benchmarks, CC BY 4.0.

Sources

Epoch AI