TurkishMMLU Contamination List
Test questions found verbatim on the Turkish web
113 of the 900 questions in the TurkishMMLU test set (12.6%) appear verbatim in a 527-million-word Turkish web corpus. This repository contains the list of those questions, the detection method and the reproduction script, so that models trained on Turkish web text can also report their TurkishMMLU score on a decontaminated subset.
Contaminated questions · strict threshold (n = 13)
12.6% of the test set · corpus scanned: 526,593,877 words
Memorisation is not capability
If a benchmark question appears verbatim in the training corpus, the model knowing that question is memorisation, not capability. The risk exists for every corpus collected from the Turkish web; most questions come from textbook and practice-exam sites, so copies of them exist online.
A third of the reported gain came from contamination
A continued-pretraining model's gain over its base measured +6.27 points on all 900 questions and +4.14 points once the contaminated questions were removed.
Exact n-word shingle matching
The same criterion used in the GPT-3 and Llama technical reports.
Normalisation
Question text is normalised: NFC, Turkish lowercasing (İ→i, I→ı), punctuation and non-digit characters replaced by spaces, then tokenised into words.
Windows
All consecutive n-word windows are extracted from each question; windows are taken from the question text and the choices (up to 5).
Scanning
The corpus is streamed with the same normalisation; if any window occurs verbatim in the corpus, the question counts as contaminated. Memory use is independent of corpus size.
| Threshold | Meaning | Contaminated | Share |
|---|---|---|---|
| n = 13strict | Chance matches are practically impossible; the standard criterion. | 113 | 12.6% |
| n = 8loose | Sensitive to partial or paraphrased leakage; false positives possible (boilerplate sentences). | 252 | 28.0% |
Corpus scanned: 526,593,877 words of Turkish web text (the first 4.3 GB in stream order). The answer key is not text and is not scanned. The list is a lower bound: paraphrased questions, or ones leaking in fragments shorter than 13 words, escape the strict threshold.
Contaminated questions by subject
n = 13 · 113 questions
Index and hash only
Each entry carries the position in the source dataset, the SHA-256 hash of the question text and the subject label. Even if the dataset's ordering changes, the question can be matched by hash.
indexPosition in Hugging Face AYueksel/TurkishMMLU, config All, split test.
sha256_soruSHA-256 of the question field (after strip, UTF-8).
konuSubject label from the source dataset.
{
"kaynak_kume": "AYueksel/TurkishMMLU (config All, split test)",
"taranan_soru": 900,
"kulliyat_kelime": 526593877,
"kirli_13": [
{"index": 0, "sha256_soru": "e84db08c…", "konu": "Biology"},
…
],
"kirli_8": [ … ]
}Derive the clean subset in three steps
Load the list, match hashes against the dataset, filter out contaminated questions: 900 → 787.
import json, hashlib
from datasets import load_dataset
ds = load_dataset("AYueksel/TurkishMMLU", "All", split="test")
k = json.load(open("turkishmmlu_kirlilik.json", encoding="utf-8"))
kirli = {r["sha256_soru"] for r in k["kirli_13"]}
def ozet(q): return hashlib.sha256(q.strip().encode("utf-8")).hexdigest()
temiz = ds.filter(lambda ex: ozet(ex["question"]) not in kirli)
print(len(ds), "->", len(temiz)) # 900 -> 787Alongside the full-set score, report the clean-787 score and the difference between the two. A large difference means the model's corpus has seen these questions.
$ python kirlilik_denetimi.py <kulliyat.txt> <etiket>The script streams the corpus line by line. If you scan your own corpus your list will differ from ours: a question on this list may be absent from your corpus, and one not on it may be present.
Known limitations
The list is relative to our corpus. A model trained on a different slice of the Turkish web has different contamination; but the same sources appear in every slice, so overlap will be high.
Question and choice text were scanned; the answer key is not text and cannot be. Paraphrased copies escape the strict threshold; the list is a lower bound.
The strict n = 13 threshold misses paraphrased copies; the n = 8 list is provided for that reason, but contains false positives.
If the source dataset's ordering changes, index becomes invalid; use sha256_soru.
If you use this list
eCloud Tech. (2026). TurkishMMLU Kirlilik Listesi: Türkçe web külliyatında birebir geçen test soruları. https://github.com/ecloudtechnology/turkishmmlu-kirlilik
License: list and script CC BY 4.0. For TurkishMMLU itself: Yüksel, A. et al. (2024), TurkishMMLU. Contact: [email protected]
The same measurement discipline
Report your TurkishMMLU score on the clean subset too
The list and the scanning script are open under CC BY 4.0. Contact us to collaborate on Turkish benchmarking and evaluation.