Benchmark containing 2500 exam-style questions crafted by experts from various fields. LLM performance still worse than human experts.
Benchmark containing questions requiring expert-level medical knowledge. o1 outperforms upper-year medical students. All other models underperform.
Benchmark testing inference of indirect intent from non-verbal responses. LLM performance worse than humans (undergrad students)
Benchmark containing FOMC transcript sentences to detect the speaker's monetary policy stance. LLM performance worse than humans (undergrad economic students). Best LLM's accuracy 0.61 vs. 0.89 human.
Benchmark testing idiom understanding across 18 high-, medium-, and low-resource languages. Performance worse than humans (tie in Vietnamese). Best LLMs average 81%, humans at 94%.
Foundational knowledge retrieval, contextual reasoning, open-ended modeling problems. Performance worse than human experts on open-ended modeling problems; outputs evaluated by human experts (PhD+)
Benchmark testing long-context understanding of live-stream transcripts via retrieval, reasoning and hybrid tasks. Overall performance worse than humans (students)
Benchmark tests general knowledge, financial domain knowledge, and financial reasoning errors in 9 scenarios. Performance worse than humans (sophomore finance students)
534-item NLI benchmark on inferring the intended meaning of phrasal constructions. Performance worse than humans for understanding syntactically identical but semantically distinct constructions
300-sample benchmark testing whether reasoning models obey six user constraints during reasoning. Trivial tasks for humans but models fail to obey during reasoning
400 expert-curated questions across eight Chinese dynasties test evidence-based historical reasoning. Performance worse than open-book historians (20 history professors and PhD)
8-subtask benchmark testing lexical comprehension and generation, with word-translation coverage across 2,700+ languages. Poor performance especially for low-resource languages
400-item benchmark testing implied meanings of identical questions across varying contexts. Performance worse than humans (62 annotators)
Benchmark includes Korean–English code-switched prompts cover content understanding and manipulation tasks. Defaults to non-English outputs (human verification)
1,290 Japanese–Chinese homographs across word meaning, word meaning in context, and translation tasks. Overall performance worse than humans (students)
9,329-instance benchmark testing prompt brittleness across offensiveness, politeness, irony, and emotion classification. Performance worse than humans (for offensiveness and emotion classification)
Filipino-centric benchmark of 27 LLMs across reading comprehension, translation, cultural knowledge, and text classification, finding pervasive weaknesses even in SEA language models.
Extends LMentry to 9 typologically diverse languages; open-weight LLMs lag humans, with native speakers near-perfect accuracy and even beginner Basque learners outperforming LLMs.
Parallel 29-language benchmark (11,829 identical questions/language) evaluating 36 LLMs; reveals large, systematic performance gaps between high- and low-resource languages on the same questions.
49,280 questions on the geography, culture, and history of 9 countries, in 7 languages. Models show greater knowledge of cultural information in English than in the culture's dominant language.
LLMs exhibit insufficient temporal referent consistency (expecting identical responses across absolute and chronological temporal references, which rely on implicit temporal information.
1,626-scenario pragmatics benchmark with a human baseline. LLMs lag human performance by 15-20 points at detecting and maintaining emotion shifts from contextual (non-lexical) cues.
11 off-the-shelf LLMs show strong multiple-choice cultural knowledge on Persian literary allusion recognition but a sharp drop in open-ended, indirect cultural reasoning.
LLMs struggle with "narrative subtleties such as irony, implication, and emotional reversal" in Japanese short-short stories; "overall [LLM] accuracy remained notably lower than the human baseline"
Evaluation of LLMs and humans on 100 representative Bangla idioms (selected by linguists). No model surpasses 50% accuracy, while humans (15 Bangladeshi students) score 83.4%
Task: annotating data of conversation interruptions to signal trouble. LLMs rely on lexical patterns rather than conversational context and construct reasonable-sounding but misleading narratives.
6 QA datasets with paraphrastic passage variants show LLM semantic understanding degrades sharply as phrasing drifts from pretraining-like text, unlike human performance.
1,000-puzzle spatial pathfinding benchmark, performance: 98.0% humans vs. 15.8% for the best reasoning model (zero-shot); reasoning traces show models make spatial-logic errors.
State-of-the-art LLMs fail at meta-reasoning (locating the first error in a stepwise solution) even with the reference solution visible.
Humans solve interactive multi-turn code-breaking benchmarks nearly perfectly while LLMs struggle dramatically (best human on hard puzzles: 100% accuracy, best LLM tested: 13% accuracy)
10 tasks with mismatched transcript claims and speaking style to test paralinguistic understanding; humans get 80.9–88.7% accuracy, while models score much lower with highly variable task rankings.
47 tasks spanning perception and reasoning tasks across linguistic and paralinguistic domains. Humans average 89.72% accuracy, versus 16.11–60.68% for models, with variable per-task rankings.
27 tasks across speech, music, and environmental sounds; humans average 82.17% on speech tasks versus 64.86% for the best model, with perceptual errors dominating model failures.
The paper shows that SpeechLLM are not good judges for speech translation.
Machine translation benchmark that curates inputs that break frontier LLMs, using robust pass/fail evaluation. Users can contribute examples in any language, across multiple modalities.
Best model covers only 0.725 of salient arguments (ARCscore) in 1,049 legal opinions vs 0.982 for human reference summaries; omissions dominate.
On the same 144 conditionals humans rated, LLMs integrate probability and relevance less systematically; mean rating of supporting conditionals is 0.19 for humans vs 0.01-0.16 for models.
Across 2,707 dialogues, models over-converge on the user's style relative to the human turn they replace in 62.5% (utterance length) and 79.2% (token novelty) of settings.
On 30 verb pairs rated by 14 Hindi-English bilinguals, no model beats chance at predicting native-speaker verb preference (best macro-F1 0.33 vs 0.27 random).
Labelling the discourse relation between two spans (DISRPT 2025: 17 labels, 38 datasets, 16 languages), the best model gets 56.8% accuracy on its strongest framework vs 72.0% for supervised systems.
TVShowGuess: name the speaker of each line of sitcom dialogue. Swapping the character names in the text for unfamiliar ones drops the best of 3 models from 78.2% to 32.0% accuracy; 3 people scored 47.1-84.4% on 100 swapped items.
Picking the correct Egyptian Arabic idiom to complete a context sentence, the best of 22 models scores 85.33% accuracy on 150 idioms; the model average is 64.45%, vs 78.52% for explaining them.
Interpreting 48 newly written modern Chinese poems, GPT-4 picked the poet's own most poetic line in only 20 of 48; the six poets scored its interpretations 73.75-88.75/100 on eight other dimensions.
Predicting how far human annotators disagree on a label, the best prompted LLM correlates 0.461 with true label variance (VarCorr, higher better) on 1,000 Gab hate posts; fine-tuned DeBERTa-V3 0.554.
Judging whether a reason explains an assertion, 21 models plateau at MCC 0.55 over 3,008 school exam items; on the 4-way version Qwen3-32B scores 73.2%, while two expert teachers matched the gold labels on 96% of 300 items.
In 900 CommonsenseQA/SIQA/PIQA items, disabled annotators marked those whose gold answer no longer holds if the person is disabled. Asked to flag the same items, none of 3 models beat 0.59 balanced accuracy (chance 0.50).
Assigning lexicalization-profile labels to six defective nouns in Italian, Russian and English, LLM matches a linguist's gold on ~600 usages at 0.787 Jaccard overall, only 0.663 on sense relations.
Writing text that implicitly conveys a signal without naming it: models beat human-written text on emotion and genre, but the best reaches 0.64 on writing skill and 0.92 on political slant against human 1.00 (expressivity score).
Six NLP tasks with items minimally rewritten (negation, dialect, style) and their labels re-checked by hand: GPT-5, the steadiest model, flips correctness on 5.3% of pairs, 16.2% under negation (unrobustness, lower better).
Answering the 1,954-item SocialIQa dev set, the best model reaches 81% accuracy against 87% for humans; 576 items (29.5%) are flawed, and cleaning them alone lifts GPT-o1 to 92%.
Explaining why a post is hateful, GPT-4o scores 0.677 quotation faithfulness on 1,370 HateXplain posts and Gemma-27B 0.365 (0-1, higher better); 3 expert moderators validated the metric on 600 explanations.
Deciding whether a word is extremist in-group slang, the best model scores 80.0 F1 on 3,050 Incels candidate words and 64.4 on 3,050 Stormfront ones, against expert sociologists' labels.
Medical exam questions missing a clinically essential fact, with an abstain option: annotators marked 38 of 49 as ones a clinician should not answer; Qwen abstained on just 5, GPT models on none.
Writing teacher-style feedback on 603 real student maths solutions, the best model scores 1.82 of 3 on expert rating by three maths-education teachers, whose own feedback is the gold reference.
Judging 600 human-chatbot dialogues in six languages for eight failure types, the best LLM judge stays under 0.4 F1+ on empathy and common sense, below both human annotators on all but one category.
Deciding whether two events in a news document are causally linked, the best off-the-shelf LLM averages 72.8% overall accuracy on 4,800 balanced human-annotated pairs, where guessing scores 50%.
1,617 Spanish, Italian and Portuguese immigration headlines labeled for hyperpartisanship, conspiracy and stance; best macro-F1 against nine annotators' labels is 0.71, 0.83 and 0.61, and no LLM matches their agreement.
Predicting how 710 MEPs voted on 47 European Parliament proposals from persona prompts, the best model scores 0.793 weighted F1 on 27,770 real votes; always predicting FOR scores 0.666.
Picking the grammatical sentence in 1,761 Quebec-French minimal pairs, even a 72B model scores 60.32% on stranded prepositions against 84.13% for 12 native annotators (accuracy; chance 55.56%).
Four creativity metrics on human-labeled sets; the best LLM judge ranks accepted ICLR papers above rejected in 9/14 groups (vs. 12 for humans) and reaches 0.54 accuracy over 1,026 papers.
Identifying which of four references a masked citation points to. GPT-5 scores 0.92 on hard items versus 0.76 for the best open model and 0.85 for PhD experts.
Sorting cards by a rule that shifts unannounced (102 trials, WCST): best model 0.53 accuracy vs a 0.77 healthy-adult norm cited from Barceló et al. 1997; on incongruent flanker trials 40-60% vs a cited 93-96%.
Judging 1,618 real-world moral dilemmas against 51,776 human judgments, LLMs invoke a narrower set of values: mean normalized value entropy 0.46 vs 0.57 for humans (higher = more diverse).
Picking a sentence's standard negation over local-negation, contradiction, paraphrase decoys: GPT-4.1 0.936 zero-shot, best open model 0.761; humans (n=11, 50 items) mean 0.78, median 0.82, best 0.96.
1,584 WinoMT translations yield 45–87% gender accuracy; professional translators reach 93.2% Cohen’s kappa on 100 cases. Only 1 of 4 systems preserves uncertainty when gender is unresolvable.
Rating whether rights-limiting actions in 1,152 UDHR scenarios are acceptable, nine of eleven models diverge in open-ended answers from the 2.86 mean of four annotators on 20 English scenarios (5-point endorsement).
Several level math reasoning benchmark extended to 18 underrepresented languages. For low levels, models are good, further: heavily drop for even English and other languages.