Big Bench Audio

Measures: 1,000 spoken reasoning questions (250 each: formal fallacies, navigate, object counting, web of lies) adapted from BIG-Bench Hard; the model answers in audio and an LLM judge marks each answer right or wrong; accuracy is over all 1,000 questions.

Maintainer: Artificial Analysis
Introduced: 2024
Status: Active
Saturation: Saturated
Typical provenance: Independent
In scoring set: No

Official leaderboard

Leaderboard

Show independent results only

Every result we hold, whoever ran it. Only independent and third-party rows count towards a score.

No results to show.