Call for Papers
WLCD 2027 – The First Workshop on What LLMs Can(not) Do
WLCD will be a workshop at COLING in Macau. Our goal is to keep track of the abilities of state-of-the-art Large Language Models (LLMs) in comparison to humans, and we are looking for contributors! At the workshop, expect a program full of invited talks, posters, discussions, and ideas.
Read below for details on submitting a paper to the workshop. Additionally, if you have a relevant paper that is already published from 2025 or later and would fit our survey table, check out the Contribute page.
Requirements
The most important requirement is that the paper compares LLMs to humans. This can be done in a variety of ways. The best of course is having a human baseline, but this is not always feasible. We also accept evaluations which are annotated by humans, such as exam questions crafted by an expert. The answer should be objective, so that the LLMs' answers can be fairly evaluated. Evaluations may also be considered trivial, such that a reasonable human may be expected to get 100% accuracy (e.g., counting the r's in strawberry).
For this year, we are limiting the scope to works that satisfy these 3 requirements:
- Works on out-of-the-box LLMs, i.e., evaluation requires no additional fine-tuning.
- Evaluations are on natural language, including text and/or speech.
- Evaluation tasks do not require tool usage, which means no RAG or knowledge graphs. If it is not clear if a closed LLM is using a tool to solve a task, this is allowed, but should be mentioned in the paper.
Feel free to email us at contact@what-llms-can-not-do.org if you're unsure whether your idea meets these requirements.
Paper Format
We accept the following paper formats:
- Long papers
- Short papers
- Update papers
All formats can choose to be either archival or non-archival. Long and short papers are your standard 8- or 4-page papers in ACL format, following instructions from the template.
Update papers are 2–4-page papers that update the findings of older works with newer models. Have you tried to replicate existing evaluations with SOTA models and found that newer LLMs perform better, similarly, or even worse than previously tested models? This is the place for you. The format is ACL style, with shorter sections overall, including an explanation of the original paper. Additional analysis is welcome, but not required.
We will accept pre-reviewed submissions from ACL Rolling Review (ARR) as well as independently on OpenReview.
Timeline
All deadlines are at 23:59, Anywhere on Earth.
- Paper submission deadline
- February 15, 2027
- Notification of acceptance
- March 5, 2027
- Camera-ready paper due
- March 12, 2027
- Workshop
- Full day, one of May 9–14, 2027
