TurkMorfBench
A Turkish morphology benchmark: not a score, a diagnosis.
TurkMorfBench measures how well a language model knows the Turkish suffix system and tells you which rule it failed. 466,517 items, 14 buckets, wug-stem controls, tokenizer analysis. Gold answers are generated by a rule engine and never come from a language model at any stage.
Items · full tier
14 buckets · core tier 1,836 items · frozen release v3.8.0
A single accuracy number cannot say which rule is missing
The output is a diagnosis, not a score: it breaks down by stem type, bucket, stem-final sound and the rule that was violated. The report below comes from a real 260-item run of an open model served over an OpenAI-compatible endpoint.
Run against local weights, the report adds a tokenizer breakdown: accuracy split by whether the vocabulary kept the stem intact or fragmented it, and the measured cost of fragmentation in points.
ZORUNLU SEÇİM 161 / 260 %61.9 GÖVDE TÜRÜ gerçek gövde (real stems) 124/179 %69.3 uydurma gövde (nonce/wug stems) 37/81 %45.7 KOVA (bucket) istisna_uyum_kirici 0/12 %0.0 ek_zinciri 4/15 %26.7 cati 5/11 %45.5 ad_cekimi 33/48 %68.8 fiil_cekimi 10/10 %100.0 SON SES (stem-final sound) t 6/18 %33.3 ç 4/10 %40.0 p 6/12 %50.0 ünlü (vowel) 28/37 %75.7 HANGİ KURAL BİLİNMİYOR (which rule was violated) uyum (vowel harmony) 28 %28.3 iyelik_unlu (buffer vowel) 22 %22.2 zamir_n (pronominal n) 10 %10.1
Fourteen buckets, each measured separately
Turkish is agglutinative and suffix selection follows strict rules: two-way and four-way vowel harmony, consonant assimilation, consonant lenition, buffer consonants. On top sit lexical exceptions (burun → burnu, kalp → kalbi, ret → reddi) and pronunciation-driven suffixes (TCDD'yi, 2026'da).
| Bucket | Items | What it tests |
|---|---|---|
| ad_yuva | 283,360 | plural + possessive + case stack, pronominal n |
| ad_cekimi | 166,424 | 7 cases, harmony, assimilation, lenition, buffer consonants |
| ek_zinciri | 9,544 | depth 1 to 8, suffix order |
| fiil_cekimi | 1,792 | 7 tense/mood, negation, aorist exceptions |
| yapim_eki | 1,477 | -lIk, -CI, -lI, -sIz, -sAl, -lAş, -lA |
| istisna_ozel_ad | 1,200 | apostrophe, no lenition (Sinop'a) |
| istisna_birlesik_isim | 896 | buzdolabına |
| cati | 728 | passive, reflexive, reciprocal, causative |
| istisna_sayi | 507 | suffix follows pronunciation (2026'da) |
| istisna_kisaltma | 304 | suffix follows pronunciation (TCDD'yi) |
| istisna_unlu_dusmesi | 177 | burnu, aklı, nakdi · TDK-verified |
| istisna_uyum_kirici | 71 | kalbi, rolü, kıraati · TDK-verified |
| istisna_ikizlesme | 35 | reddi, tıbbı, zıddı · TDK-verified |
| istisna_kaynastirma_istisna | 2 | suyu, neyi |
| Total | 466,517 |
Not noise: each one breaks exactly one rule
Every distractor violates a single rule and is named after it. A model that picks içtimaiyatlarıda does not know the pronominal n; one that picks içtimaiyatlarınde does not know vowel harmony. This is what makes the benchmark diagnostic rather than a scoreboard.
The hucre field carries the phonological environment (final vowel, final-sound class, syllable count, internal harmony); bayrak carries lexical irregularities (LastVowelDrop, Doubling, InverseHarmony, CompoundP3sg).
Options are shuffled and the gold sits at a random position (altin_sira records which). The shuffle is derived from the item id, so an item always gets the same order and cross-version comparisons stay intact. This was not the case before 3.2.1: the gold was always first, and under a protocol that letters the options, always answering A would have scored 100%.
{
"kimlik": "6c4f2594d126",
"kova": "ad_yuva",
"gorev": "cog-3t-bulunma",
"govde": "içtimaiyat",
"altin": "içtimaiyatlarında",
"celdirici_tur": ["zamir_n", "uyum"],
"celdirici_bicim": ["içtimaiyatlarıda", "içtimaiyatlarınde"],
"secenek": ["içtimaiyatlarıda", "içtimaiyatlarında", "içtimaiyatlarınde"],
"altin_sira": 1,
"gercek": true,
"hucre": ["a", "t", 3, "kirik"],
"bayrak": []
}The 33 cells real words cannot reach
The axes that determine suffix selection are crossed first and stems are then generated to fill those cells: final vowel (8) × final-sound class (8) × syllable count (3) × internal harmony (2) = 320 cells.
33 of those cells contain no real Turkish word at all. A benchmark built from real words is structurally blind there. We measured it: models score 22-23 points worse in exactly those cells, even after matching phonological composition.
Rule knowledge, or lexical neighbourhood?
Inflecting a real word can be memorisation: a model has seen the form kitabı tens of thousands of times. A nonce stem (zakak, vısep, sövot) appears in no corpus, so only the rule can produce the right form.
Measured gap: models score 22-32 points lower on nonce stems than on real ones.
Never from a language model, at any stage
Published open Turkish morphological analysers measure 38-72% accuracy; using one as ground truth would inject 28-62% error. So arbitrary words are not analysed — items are generated whose gold form follows by construction.
Rule engine
A phonological engine applying vowel harmony, assimilation, lenition and buffer consonants. It carries 282 hand-written gold assertions, all passing.
Zemberek dictionary
Lexical irregularity flags (which stem lenites, which drops a vowel) come from the Zemberek dictionary (Apache-2.0). Its analyser is not used.
TDK Güncel Türkçe Sözlük
All 283 items in the three exception buckets were checked character-for-character against TDK. If the dictionary gives no form for a stem, or more than one, that stem is excluded.
In the human-ceiling study native speakers scored below chance on the inverse-harmony bucket. When native speakers do worse than guessing, the gold is what needs checking, not the speakers. It found three systematic errors — applying lenition where the dictionary forbids it (bidadi, correct bidati), missing inverse harmony (aczı, correct aczi), and failing to voice a doubled stop (retti, correct reddi) — all from Zemberek flags being incomplete against TDK. The correction covered 147 dictionary entries and was applied not just to the exception buckets but to every suffix stack in every bucket.
The same items are put to native Turkish speakers
This is still a preliminary reading: the sample is small, the interval wide, and the figure will be updated as more raters finish. The interval uses phonological-cell clustered bootstrap; computed at item level it would read 74.6–81.8%, because items sharing a cell are not independent observations.
[70.4% – 83.2%]
6 valid raters · 560 judgments · collection ongoing
| Slice | Accuracy | n |
|---|---|---|
| Real stems | 79.5%[75.8% – 82.9%] | 531 |
| Nonce stems | 55.2%[34.5% – 74.1%] | 29 |
Nothing is being claimed yet about human performance on nonce stems: the previous version's "humans drop 12 points" claim has also been withdrawn. The 24.3-point point estimate here rests on dozens of judgments and the interval permits no conclusion in either direction.
The 88.2% [83.1 – 91.8] reported in the previous version has been withdrawn. There are two reasons, both on the benchmark's side: that figure rested on 3 raters and the distribution widened at six (valid raters range from 96.7% down to 53.2%); and 257 of the 520 items in that measurement's pool were not in the current benchmark data. The pool was rebuilt from current data on 24 September 2026.
Rater screening is the real defence of the measurement. Three criteria are fixed in advance: at least 10 answers; a median under 3 seconds per question means the question was not read; accuracy that fails to beat the rater's own chance level on a one-sided binomial test at α = 0.001 carries no information. Six of twelve raters were screened out, five on the speed criterion — one answered 111 questions in 36 seconds. All five speed failures were also at chance level.
On 25 September 2026 AIGENCY V4 was run through the same interface as the human study, answering all 515 items across 50 sessions. The speed criterion separated those sessions from the human data; they are flagged as model sessions in the database and do not enter the human-ceiling figure. On those same 515 items AIGENCY V4 scored 69.9% and valid humans 79.1%.
Seventeen models, one pinned script, clustered confidence intervals
3.6 core (1,836 items), the harf rule, cell-clustered 95% CI, bf16, option order frozen. The TurkishMMLU column was measured with the same pinned script on 652 clean questions (five-way, chance 20%). Erk models were improved using this benchmark's diagnostic output; the other models were measured as published.
| # | Model | Developer | Size | TurkMorfBench | 95% CI | TurkishMMLU |
|---|---|---|---|---|---|---|
| 1 | Erk-14B(v2) | eCloud Tech. | 14B | 80.0% | 75.8% – 88.3% | 66.9% |
| 2 | turkish-gpt2-large | YTÜ COSMOS | 0,8B | 79.1% | 76.9% – 81.8% | 18.4% |
| 3 | kanarya-2b | Asafaya | 2B | 78.4% | 76.4% – 83.3% | 22.9% |
| 4 | Erk-32B(v2) | eCloud Tech. | 32B | 76.5% | 71.0% – 87.3% | 71.5% |
| 5 | Kumru-2B-Base | VNGRS | 2B | 76.2% | 72.6% – 77.8% | 17.0% |
| 6 | Kumru-2B | VNGRS | 2B | 76.2% | 74.0% – 78.7% | 19.6% |
| 7 | Turkcell-LLM-7b-v1 | Turkcell | 7B | 76.0% | 74.2% – 80.5% | 26.4% |
| 8 | Erk-14B(v1) | eCloud Tech. | 14B | 75.4% | 70.0% – 86.0% | 67.2% |
| 9 | Qwen3.8-27B(taban) | Alibaba | 27B | 75.4% | 71.3% – 83.9% | 70.2% |
| 10 | Turkish-Gemma-9b-v0.1 | YTÜ COSMOS | 9B | 75.2% | 69.8% – 85.7% | 60.3% |
| 11 | Erk-32B(v1) | eCloud Tech. | 32B | 74.9% | 69.2% – 86.1% | 71.0% |
| 12 | wiroai-turkish-llm-9b | WiroAI | 9B | 74.6% | 70.4% – 83.1% | 45.7% |
| 13 | Trendyol-LLM-7B-chat-v4.1.0 | Trendyol | 7B | 73.5% | 71.0% – 79.2% | 50.0% |
| 14 | Qwen3-32B(taban) | Alibaba | 32B | 73.4% | 67.8% – 84.3% | 67.6% |
| 15 | Turkish-Llama-8b-v0.1 | YTÜ COSMOS | 8B | 72.0% | 68.8% – 79.0% | 35.0% |
| 16 | Turkish-Llama-8b-Instruct-v0.1 | YTÜ COSMOS | 8B | 70.2% | 65.4% – 79.9% | 42.2% |
| 17 | Trendyol-LLM-7b-chat-v1.0 | Trendyol | 7B | 67.6% | 65.3% – 73.3% | 29.9% |
The top places sit in a statistical tie band; two models are compared by the clustered bootstrap of their paired difference, not by whether their intervals overlap. Morphology does not scale with parameter count: 0.8-2B models at chance on TurkishMMLU sit in the same band as 32B models on morphology. The benchmark therefore does not answer "which model is good" on its own; it must be read alongside a general-capability measure.
The benchmark's own errors — written down as they are found
3.6.0 · harmony under vowel drop
In vowel drop (aciz → aczi) the harmony of a vowel-initial suffix was taken from the last vowel of the remaining stem instead of the dropped vowel. The TDK audit had "fixed" this with a flag, but the flag leaked into -lAr: acizlarında, correct acizlerinde. Affected: 34 stems, 136 wrong gold forms in ad_yuva. The engine was fixed, the flag search re-run (501 of 501 TDK gold forms reproduced), and the benchmark regenerated.
3.5.0 · the ek_zinciri bucket
Five models scored exactly 0% at depths 2/3/5/7 and ~100% at 6/8; that cannot be a model property, it was an artefact of the bucket design. The old distractors were "one step short" and "one step long" forms — both grammatical. Distractors are now rule violations, and depth-1 items entered the bucket (8,351 → 9,544).
3.7 · scoring frozen
The scoring code was pinned in 3.7.0 and 3.8.0 shipped as the frozen release (Production/Stable on PyPI). The ek_zinciri bucket, once corrected, went to the ceiling instead: ≥96% for 13 of 15 models on the 3.6 core, so it no longer discriminates. The next data release will use grammatical chains that disagree with the context as distractors.
These fixes break cross-version comparison: ad_yuva, ad_cekimi and istisna_uyum_kirici cannot be compared with 3.5.x, and ek_zinciri cannot be compared with 3.4.x. The other buckets are unchanged.
466,517 items are not 466,517 independent observations
Items are not generated independently: one stem yields dozens of items across the plural × possessive × case cross, and one phonological cell binds hundreds of stems to the same rule. If a model misses lenition on the stem kitap, it misses it across most items derived from that stem. Item-level bootstrap ignores this, treats the sample as larger than it is and narrows the interval — false precision.
| Clustering unit | Clusters |
|---|---|
| Stem | 32,140 |
| Phonological cell | 441 |
The resampling unit moves from the item to the cluster, and the design effect is measured and reported on every run; the measured value ranges from 1.2× to 4.1× across models. Comparing two models by whether their separate intervals overlap is wrong — the clustered bootstrap of the paired difference is used instead (esli_fark).
Six rules, one ranking
The model is run once; for each candidate the raw sum, token count, character count and unconditional probability are stored, and six rules are derived from those three quantities. Across 15 models harf, ham and jeton give the same extremes (the top two and bottom three places are identical), with ham 1-4 and jeton 2-16 points lower. The findings therefore do not depend on the scoring rule. The design effect ranges from 1.2× to 4.1× between models.
| Rule | Picks the shortest (target 29.9%) |
|---|---|
| Mean over the whole string (old) | 47.2% · 47.7% |
| Candidate sum | 51.0% · 57.0% |
| Divided by tokens | 47.5% · 48.7% |
| Divided by characters (new) | 39.4% · 43.2% |
| Normalised by the unconditional score | 39.2% · 45.0% |
pmi and pmi_harf collapse to the 21-51 range: normalising by the unconditional probability destroys the distinction between candidates that share a stem. Those two rules are unfit for this task; they stay in the package so the reader can see it for themselves.
Core for diagnosis, full for the headline number
Runs in minutes, bucket-balanced — so that even the smallest bucket yields a measurable estimate.
The whole of the designed coverage.
The core tier is bucket-balanced, which means it is deliberately not an unbiased sample of the full set. Do not compare the two.
Forced choice is primary, free generation secondary
Forced choice
The model picks between the gold form and rule-violation distractors by log-likelihood. There is no output parsing, so format compliance cannot affect the result.
Free generation
Secondary, and it counts unreadable separately from wrong. Without that distinction one of our own measurements was off by fifty points.
Three steps from install to diagnosis
The package is on PyPI and runs against local weights or an OpenAI-compatible endpoint (vLLM, llama.cpp server, TGI). With a remote endpoint text_offset is required in the /v1/completions response; without it candidate tokens cannot be separated and the run stops rather than silently computing something else.
1 · Install and run
pip install turkmorfbench, then turkmorfbench olc --model <MODEL>. The default is the core tier and finishes in minutes; turkmorfbench bilgi prints the installed package and data version.
2 · Narrow the scope
--katman tam runs the full set, --govde uydurma only the nonce stems, --kova <name> individual buckets. --cikti rapor.json writes the per-item prediction vector to disk; --isci sets the number of parallel requests.
3 · Read it correctly
The output is a diagnosis, not a score: which rule, which stem-final sound, real or nonce. Do not compare two models by whether their separate intervals overlap; take the clustered bootstrap of the paired difference with esli_fark. The duyarlilik module tells you whether a result depends on the scoring rule.
$ pip install turkmorfbench$ $ # yerel ağırlıklar / local weights$ turkmorfbench olc --model ecloudtech/Erk-32B$ $ # OpenAI uyumlu uç / OpenAI-compatible endpoint$ turkmorfbench olc --uc http://localhost:8000/v1 --model erk --isci 16$ $ # tam katman, yalnız uydurma gövdeler, JSON rapor$ turkmorfbench olc --model X --katman tam --govde uydurma --cikti rapor.json$ $ # yalnız istisna kovaları$ turkmorfbench olc --model X --kova istisna_unlu_dusmesi istisna_ikizlesme
from datasets import load_dataset
d = load_dataset("ecloudtech/TurkMorfBench", "cekirdek") # 1.836
d = load_dataset("ecloudtech/TurkMorfBench", "tam") # 466.517from turkmorfbench.istatistik import kume_bootstrap, esli_fark, tasarim_etkisi
from turkmorfbench.duyarlilik import bilesen_topla, karsilastir, rapor
s = kume_bootstrap(maddeler, dogru, anahtar="hucre")
f = esli_fark(maddeler, a_dogru, b_dogru, anahtar="hucre")
print(rapor(karsilastir(bilesen_topla(arka_a, maddeler),
bilesen_topla(arka_b, maddeler),
maddeler, "A", "B"), "A", "B"))We know these and we write them down
Homographic stems are excluded: their inflection diverges, so the gold answer is not unique (278 stems in the general buckets).
More than half the exception stems are absent from TDK's current dictionary (archaic loans such as inhimak, meşihat, pirüpak); they were excluded in this version, so the exception buckets now measure living Turkish.
66 causative items were dropped: on monosyllabic real stems the suffix is lexical, and those not on the explicit exception list were treated as unreliable.
Exception buckets are real-stem by nature — a nonce word cannot have a lexical exception.
The human ceiling is a preliminary reading: 3 raters, 211 judgments, wide interval.
The length bias in the scoring rule was reduced but not eliminated (39-43% against a 29.9% target).
The design effect (cell-clustered over item-level interval width) varies from 1.2× to 4.7× between models; it is highest on the Erk models, meaning they are very strong in some rule cells and weak in others.
ek_zinciri scores ≥96% for 13 of 15 models on the 3.6 core and does not discriminate; the ranking without it is identical.
istisna_kaynastirma_istisna has only two items (suyu, neyi); it is not interpreted on its own.
3.8.0 is the frozen release
Data 3.6.0, scoring 3.7.0. Every change to data or scoring code is recorded with its rationale in CHANGELOG.md; figures from different data versions are not comparable. The package checks documentation, data and the published package against each other with tutarlilik.py.
Numbers measured with v2
The TurkMorfBench figures on the Erk-32B page (the 512-item v2 set: 91.43% forced choice, 59.38% generation) were measured with v2 and stand as that version's numbers. v3.6 is a different set with a different scoring rule and a different bucket structure; figures from the two versions are not comparable.
If you use this benchmark
@misc{turkmorfbench2026,
title = {TurkMorfBench: A Diagnostic Morphology Benchmark for Turkish Language Models},
author = {{eCloud Tech.}},
year = {2026},
url = {https://huggingface.co/datasets/ecloudtech/TurkMorfBench}
}Code Apache-2.0, data CC BY 4.0. Contact: [email protected]
Measure your model with TurkMorfBench
One command to install, a diagnostic report as output. Code Apache-2.0, data CC BY 4.0. Contact us to collaborate on evaluation for Turkish models.