Suggest an addition

Propose a benchmark or paper for the table. Sign in with GitHub, then submit to open a pull request under your account for review.

Criteria

  • The paper must be from 2025 or later. If the original work is older but someone has tested the same task on a newer model, submit that newer evaluation instead.
  • The paper must include a human baseline, or use objective human-annotated labels.

For now, we will not accept papers that fall under:

  • Non-natural language tasks (e.g. computer vision)
  • Tasks focused on tool use or agentic LLMs

We're looking for experts in these fields to help review submissions. If that's you, contact us.

Checking GitHub sign-in…

Paper / benchmark

KeywordsClick chips to add or remove

Evaluation details

Who is better?

Models are closed if they have no publicly available weights, open-weight if their weights are available but not their data, and open-source if their weights and training data are publicly available.

Put “Humans (trivial)” for tasks without human evaluation that are nevertheless obviously easy for humans (e.g., make this text all caps).

Sign in with GitHub to submit