What LLMs Can(not) Do

What LLMs Can(not) Do logo: a human beating a robot at rock-paper-scissors

A living survey of benchmarks that compare large language models with humans.

About

LLMs can do a lot these days, but what can't they do? Which tasks can humans still do better? We're looking to gather all of the research into this and make it easily accessible for anyone interested.

We've been accepted as a workshop for COLING 2027! See our Call for Papers. See you in Macau!

Organizers

  • Lukas Edman (TU Munich)
  • Hanna Shcharbakova (TU Munich)
  • JiWoo Hwang (TU Munich)
  • Sophie Henning (TU Munich)
  • Sara Candussio (University of Trieste)
  • Marianne de Heer Kloots (University of Amsterdam)
  • Maike Züfle (KIT Karlsruhe)
  • Kathy Hämmerl (TU Munich)
  • Vilém Zouhar (ETH Zurich)
  • Lisa Bylinina (Utrecht University)
  • Leonie Weissweiler (Leipzig University)
  • Alexander Fraser (TU Munich)

Contact

contact@what-llms-can-not-do.org