An open challenge from the International Linguistics Olympiad
Can a system reason out a
language it has never seen?
Every year, the strongest young linguists are given a few lines of a language they have never seen and asked to work out the rest using nothing but logic. IOL‑AI 2026 gives your system the same problems under the same rules, and asks one question: how close can it get to an Olympiad medallist?
until the submission deadline · 26 Jul 2026, 23:59 UTC
The mission
A reasoning benchmark you can't memorise your way through.
Linguistics problems cannot be solved from memorised facts. Each one is built on a language the solver has almost certainly never seen, so the only way to the answer is to reason from a handful of examples. That makes it a test of reasoning rather than recall.
IOL‑AI 2026 turns the International Linguistics Olympiad into a public, reproducible benchmark for that kind of reasoning. Your system faces the same contest the human contestants faced. Entries are scored automatically, and the strongest are reviewed by hand by the jury, on equal footing with the students. Everything runs in the open: public entries, open method, public leaderboard.
First principles only
No language can be looked up. Every answer is deduced from the data in the problem.
Human-comparable
The same problems, point values, and scoring the IOL uses for its own contestants.
Open and reproducible
Submit a public Hugging Face repository and a script. Anyone can rerun it and verify the result.
The task
Self-contained logic problems, no outside knowledge required.
Each problem gives a small amount of data from a language the solver has almost certainly never seen, plus the hints needed to solve it. From those examples, a solution works out enough of the grammar and vocabulary to translate new forms, fill in blanks, answer multiple-choice items, and correct mistakes.
You submit a Hugging Face repository containing a script.py. The platform
runs your script on a hidden test set from the IOL individual contest and scores the
answers it produces. Everything needed is in each problem's context, so no
prior knowledge of the languages is required.
Examples of the type of task your model could face:
Translation
CONTEXT
Here are sentences in Hakhun with English:
ŋa ka kɤ ne | Do I go?
nɤ ʒip tuʔ ne | Did you sleep?
[…]
QUERY
Translate into English:
nɤ ʒip ku ne
ANSWER
Do you sleep?Number transliteration
CONTEXT
Squares of 1–10 in Ndom, in arbitrary order:
mer an thef abo thonith
nif thef abo tondor abo mer abo thonith
[…]
QUERY
Write in numerals:
nif ithin abo ithin
ANSWER
111Scoring
Get it exactly right and get close. You need both.
Exact match
Is the answer exactly right? Points-weighted using the official IOL point values, so harder, heavier items matter more.
chrF
Character n-gram overlap with the correct answer, giving partial credit for near-misses so the leaderboard is smooth rather than a step function.
Final score
The geometric mean of the two, on a 0 to 1 scale. Because it is a geometric mean, fuzzy overlap alone or a few exact hits alone will not get you far.
The public leaderboard scores a subset of items on every submission. The private leaderboard covers the rest and is revealed at the deadline. Pick your 2 final submissions before then.
The data
Hidden, self-contained, and mounted at evaluation time.
The test set is the IOL 2026 individual contest, reformatted into self-contained
problems in the Linguini format
(Sánchez et al., 2024). It is hidden: your script.py reads it at evaluation
time from /tmp/data/test.csv, which the platform mounts before your script
runs. There is no separate training set, because each problem carries all the data needed
to solve it.
Each row of test.csv is one complete problem: a block of related numbered
items that share the same data and instruction. It is not a single item. Your script
produces one answer per numbered item, returned as a list.
| Column | Description |
|---|---|
id | Unique problem id, e.g. 012023020100. Echo it back unchanged in your submission. |
context | The full problem statement: the bilingual data, hints, and language meta-information. Everything needed to solve the problem is here, and it is shared by every item in the row. |
query | The instruction followed by the numbered items to answer, e.g. Translate into English: then 17. …, 18. …, 19. …. Answer every item, in the order it appears. |
work_lang | The language the problem is presented in (FLORES code, e.g. eng_Latn). This is the language of the instructions, not necessarily the language of your answer. |
task_lang | The problem language (FLORES code, e.g. apu_Latn = Apurinã). |
task_type | The kind of task, for example translation, fill_blanks, match_letters, text_to_num, or num_to_text. The live set may include other IOL task types. |
eval_type | single or multi. With single, each item has one expected answer. With multi, at least one item accepts several equally-correct answers, for example a language with free word order where more than one ordering is valid. |
Human Evaluation Challenge (optional)
Judged by the IOL 2026 jury, on equal footing with the contestants.
Fill in the explanation field in your submission to opt into the parallel
human-judged track. It has no effect on your automatic score (explanation_rate
is reported on the leaderboard but is not part of score), but will be required
for participation in the human evaluation challenge. It should not contain a raw reasoning
trace but instead a short explanation of the answer of your model, and may contain structures
such as bullet points or a table.
We select the top 10 submissions by score that carry a valid explanation on a majority of their problems for human judgement by IOL 2026 judges, on equal footing with the contestants. Leave the column out, or leave it blank, if you only want to compete on the automatic leaderboard.
Note that the AI challenge is separate from the actual IOL: the LLM solutions will not be directly compared to contestants' solutions, and will not be part of the actual IOL event.
The challenge is simply a collaboration with members of the IOL Jury, which ensures that AI researchers can submit solutions before problems become publicly available and that expert evaluation can be done on site. Importantly, the problem set is securely managed only by members of the IOL Jury through Hugging Face and not shared directly with any participating researcher.
We hope that this collaboration will not only increase mutual awareness between the fields of linguistics and AI, but will also lead to an impactful research output on the linguistic reasoning capabilities of LLMs.
How to submit
A public Hugging Face model repo with a script.py that writes submission.csv.
- Put your model and a
script.pyin one public Hugging Face model repo (private repos cannot be submitted). The eval sandbox has no internet, so ship the model weights in the repo and load them from"."(the repo is your working directory at run time). - Have the script read the test set from
/tmp/data/test.csv. You cannot download it: your token is revoked before your code runs. - Have the script write
submission.csvwith one row per problem: anidcopied from the test set and apredwith your answers. Add anexplanationcolumn to enter the jury track. - Enter your model repo id in the competition Space. You get up to 3 submissions a day, and pick 2 for the private leaderboard before the deadline. Each run has a 30 minutes limit.
Working in a team? Teams of up to 3 are welcome. In the competition Space, the leader copies their team code from the “Your team” card and shares it; each teammate pastes it there to join. Any member can submit, and the team shares one leaderboard entry and one daily quota. No forms, no waiting: paste the code and you are in.
Resources (read these first). The resources for participants (Google Doc) walk you through everything, the workshop repo + Colab lets you get started on the linguini task and test your script.py before submitting, and the Submission 101 video walks through the whole process step by step.
The eval sandbox has no internet, so ship your model inside the (public) submission repo and load it from ".". This script.py uses Qwen/Qwen2.5-1.5B-Instruct, an openly-licensed model, with greedy decoding so results are reproducible. The one-time step that downloads the weights into your repo is documented on the competition Space. Because the repo is public, use a model whose license permits redistributing its weights.
import os
# The repo is the working directory at run time, and there is no network.
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"
MODEL_ID = "."
import json
import pandas as pd
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, torch_dtype=torch.float16, device_map="auto"
).eval()
df = pd.read_csv("/tmp/data/test.csv", dtype=str).fillna("")
rows = []
for _, r in df.iterrows():
messages = [
{"role": "system", "content":
"You solve International Linguistics Olympiad problems. Answer every numbered "
"item. Put each answer on its own line, in order, with no numbering and no extra text."},
{"role": "user", "content": f"{r['context'].strip()}\n\n{r['query'].strip()}"},
]
ids = tok.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt",
).to(model.device)
with torch.no_grad():
out = model.generate(ids, max_new_tokens=512, do_sample=False)
text = tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True).strip()
answers = [ln.strip() for ln in text.splitlines() if ln.strip()]
rows.append({"id": r["id"], "pred": json.dumps(answers, ensure_ascii=False)})
print(f"{len(rows)}/{len(df)} done", flush=True)
pd.DataFrame(rows).to_csv("submission.csv", index=False)
print("wrote submission.csv", flush=True)Running a bigger model? To fit one in the T4's 16 GB, both bitsandbytes and autoawq are pre-installed (there is no internet at run time, so you cannot install anything else). The simplest is bitsandbytes 4-bit: load your model with BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16) and use float16 (the T4 has no bfloat16). You can also ship AWQ-quantized weights and load them directly. The Submission tab lists every pre-installed library and version.
Final leaderboard
The official final standings, scored on the full question set (public and hidden private together). One entry per team.
Correction to the final results. A scoring configuration error introduced on the last day of the competition caused submissions evaluated on 26 July to be graded on only part of the question set, while earlier submissions were graded on the full set. Every submission has been re-scored with the official metric over the full question set. The table below shows the corrected standings, which are the official final results and replace the ranking published previously. Winner tiers were re-derived from the corrected scores using the same tier rule. We apologise to the affected teams and thank them for their understanding.
Congratulations to our winners. The leading teams are grouped into Tier 1, Tier 2 and Tier 3, with honorable mentions for the group behind them. Thank you to everyone who entered.
Each team's entry is their best selected submission scored on the full question set, or their best public one if they selected none.
| # | Team | Score | Exact match | chrF | Expl. % |
|---|---|---|---|---|---|
| Loading final results… | |||||
Public leaderboard
Best public-split submission per team, as it stood during the competition.
🏅 Join the Human Evaluation Challenge. Add an explanation column to your submission.csv with a short, human-readable explanation of how your model reaches its answers (not a raw reasoning trace). It has no effect on your automatic score, and it makes you eligible for review by the IOL 2026 jury, on equal footing with the human contestants.
| # | Team | Score | Exact match | chrF | Expl. % |
|---|---|---|---|---|---|
| Loading leaderboard… | |||||
Final snapshot, taken after the competition closed on 26 July 2026. It is no longer updated.
Timeline
- 29 Jun 2026Competition opens
- 26 Jul 2026, 23:59 UTCSubmission deadline. Select your 2 final entries
- 26 Jul 2026Private leaderboard revealed
- During the IOLHuman Evaluation Challenge reviewed by the jury
Organizing team
NLP researchers and linguistics-olympiad alumni.
Alexander Piperski
Lecturer, Stockholm University · IOL Problem Committee
Have a question? Contact us on our Google Group.
Fireside chat
Want to learn more about the challenge?
Watch the recording of our fireside chat with guests Dan Mirea (Princeton University) and Eduardo Sánchez (UCL and Meta), hosted by Julia Kreutzer (Cohere Labs). They introduce the challenge and the motivation behind it: to better understand where current AI reasoning models succeed or struggle in linguistics, a domain that remains underexplored compared to math and coding. A great primer on what it takes to succeed at the IOL.