Erk-32B
A Turkish language model, with a measurement protocol.
Erk-32B was built by eCloud Tech. on the open-source Qwen3-32B base with 1.05 billion tokens of Turkish continued pretraining and 423 million tokens of Turkish instruction tuning. Every number on this page passed a contamination audit, option rotation, forced-choice measurement and paired-bootstrap confidence intervals; the protocol, the corrected raw numbers and the reproduction scripts are open in the repository.
TurkishMMLU · clean 652 · option rotation
Base Qwen3-32B: 67.64% · Difference +3.37 · 95% CI [+0.61, +6.13]
Operating point: LoRA scale 0.75 + system prompt, merged weights
Every difference carries a paired-bootstrap 95% confidence interval from 10,000 resamples. An interval that includes zero is reported as not demonstrated under this protocol; it is never counted as a gain.
| Measurement | Qwen3-32B | Erk-32B | Difference | 95% CI |
|---|---|---|---|---|
| TurkishMMLU, clean 652, option rotation | 67.64% | 71.01% | +3.37 | [+0.61, +6.13] |
| TurkMorfBench v2 generation (512, exact match, system prompt) | 31.64% | 59.38% | +27.73 | [+22.46, +33.01] |
| TurkMorfBench v2 forced choice, total (502) | 89.44% | 91.43% | +1.99 | [−0.20, +4.18]not demonstrated |
| TurkMorfBench v2 forced choice, real words (312) | 94.55% | 96.47% | +1.92 | [−0.32, +4.17]not demonstrated |
| TurkMorfBench v2 forced choice, nonce (190) | 81.05% | 83.16% | +2.11 | [−2.11, +6.32]not demonstrated |
| Identity (30 questions, with traps, system prompt) | — | 30/30 | — | — |
Single training seed; the confidence intervals cover question sampling, not the seed. The +28 points on the generation axis are largely format compliance: the base model stays inside its thinking block and never writes the answer.
Four models under the same protocol
TurkishMMLU · clean 652 · option rotation · accuracy
The 69.7% on Erk-14B's own model card is a full-set, non-rotated, lm-evaluation-harness 0-shot measurement; the 67.18% here is the same model measured on the decontaminated, rotated set. The difference comes from the protocol, not the model.
Differences that could not be demonstrated
- No gain over the base in morphological knowledge (forced choice): the lower bounds sit just below zero, and on nonce stems there is none at all.
- Continued pretraining improved Turkish knowledge; it did not measurably change the suffix system.
- The +28 points on the generation axis are largely format compliance; the base model stays inside its thinking block and never writes the answer.
- The +3.73 morphology advantage reported with the first benchmark version rested on 18 wrong gold answers and was withdrawn.
No number is reported before it passes these filters
Most raw measurements turned out to be measurement artefacts. The full protocol, the measurement log and the corrected numbers are open in OLCUM-PROTOKOLU.md across eight sections.
Contamination audit
113 of TurkishMMLU's 900 questions appear verbatim in the continued-pretraining corpus (13-word window). Those and the 150-question tuning set are excluded: clean 652.
Option rotation
Every question is measured with rotated answer order; letter and position bias is removed.
Forced choice vs. generation
Format compliance is separated from knowledge: a +28-point gap in generation becomes +2.0 in forced choice, not demonstrated.
Auditing the benchmark
18 wrong gold answers were found in the first version of the morphology benchmark; every number was re-measured with the corrected v2.
Paired bootstrap
10,000 resamples; a difference whose interval includes zero is written as not demonstrated.
The log of corrected raw numbers has twelve entries; none was reported from outside, all were found in our own audit before publication.
How it was trained
The base model is not ours and is stated everywhere. 53.5% of code-bearing answers in the instruction corpus were broken by machine translation; the cleaning and repair scripts are in the repository. Benchmark contamination in the instruction corpus: 0/900 at the n=13 threshold.
- Base
- Qwen3-32B (Apache 2.0)
- Continued pretraining
- 1.05B Turkish web tokens · LoRA r=256 α=512 · 32,000 steps
- Instruction tuning
- 423M tokens, 373,387 examples (cleaned) · 2 rounds
- Extra round
- Identity 6,000 + morphological error correction 40,000 examples · 1,673 steps
- Hardware
- 4×A100-80GB, FSDP, bf16 · ≈37,000 core-hours
- Released weights
- Merged at LoRA 0.75 (fp32 multiply, single bf16 rounding); equivalence with the LoRA path verified with the same evaluation script
- GGUF
- Q8_0 34.8 GB (NLL +0.04%) · Q4_K_M 19.8 GB (NLL +0.85%); identity 6/6 for every quant
How 0.75 was chosen
Because LoRA is additive, the scale is a post-training dial. Three axes were measured at twelve points; since identity is supplied by the system prompt, the comparison was made between prompted points. In generation, 0.75 + prompt beats 0.40 + prompt by +10.35 [+5.47, +15.43], significant; on the benchmark, 0.40's advantage of +1.99 [0.00, +3.99] cannot be demonstrated. Because an interval that includes zero counts as not demonstrated under this protocol, 0.75 was chosen; the 0.40 and 0.80 rows stay in the report.
Identity is not baked into the weights
Identity is supplied by the default system prompt in the chat template. If the user provides their own system message, the default does not apply. The prompt names the base model explicitly; on the 30-question identity test with trap questions the result is 30/30.
Sen Erk-32B'sin: eCloud Tech. tarafından, açık kaynaklı Qwen3-32B temel modeli üzerine Türkçe devam-eğitimi ve talimat ayarıyla geliştirilmiş 32 milyar parametreli bir Türkçe dil modelisin. …Quantized weights for llama.cpp, Ollama and LM Studio
Every measurement and the protocol on the main model card also apply to the GGUF files; the cost of quantization was measured.
| File | Quant | Size | NLL vs. BF16 | Identity (6 q.) | Recommended |
|---|---|---|---|---|---|
| Erk-32B-Q8_0.gguf | Q8_0 | 34.8 GB | +0.04% | 6/6 | when quality comes first |
| Erk-32B-Q4_K_M.gguf | Q4_K_M | 19.8 GB | +0.85% | 6/6 | 24 GB GPU / 32 GB RAM |
NLL is the mean negative log-likelihood on 6,132 tokens of Turkish text; BF16 GGUF reference 1.5185. Identity was measured on six questions (traps included) with the default system prompt from the chat template.
$ llama-cli -m Erk-32B-Q4_K_M.gguf -cnv -p "" \ --temp 0.6 --top-p 0.95 -c 8192
$ cat > Modelfile <<EOF FROM ./Erk-32B-Q4_K_M.gguf PARAMETER temperature 0.6 EOF$ ollama create erk-32b -f Modelfile && ollama run erk-32b
Qwen3's hybrid thinking mode is preserved; append /no_think to the message to switch it off. Deployment options for LM Studio, vLLM, Jan and Docker are on the model card.
A few lines with Transformers
The model loads directly from Hugging Face; the chat template carries the default identity prompt. Use /no_think to disable thinking mode.
from transformers import AutoModelForCausalLM, AutoTokenizer
m = "ecloudtech/Erk-32B"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, torch_dtype="auto", device_map="auto")
msgs = [{"role": "user", "content": "Türkçede ünsüz yumuşaması nedir? /no_think"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=300)[0][ids.shape[1]:], skip_special_tokens=True))Erk-32B at work
The video was recorded at LoRA scale 0.40 before the operating point was finalised; the 73.16% on screen is that point's pre-extra-round measurement. The published number is in the table above. The response stream and terminal interface are identical. 67 seconds, 1440p, no audio.
Known limitations
Single seed; variance across seeds was not measured.
Generation quality, long context, code and reasoning were not measured.
Identity depends on the system prompt; deployed without it, the model states the base model's identity.
No gain in morphological knowledge could be demonstrated; the generation-axis difference includes format compliance.
The base model's corpus cannot be audited; the comparison assumes both sides are contaminated to an unknown degree.
Report, scripts and prediction vectors, all open
License Apache 2.0, same as the base model. Contact: [email protected]
OLCUM-PROTOKOLU.md
Full technical report; eight sections including the measurement log and corrected numbers.
betikler/
Training, cleaning, evaluation, scanning, bootstrap, merging and GGUF scripts; 34 files, llama.cpp version pinned.
sonuclar/
Per-question prediction vectors; 17 JSON files. Paired comparisons can be redone without running the model.
Erk-32B-GGUF
Q8_0 and Q4_K_M quants, validation summary and the card for the quantized builds.
The open benchmarks the measurement rests on
Training and evaluation ran on the high-performance computing resources provided under the AI EDIH Türkiye programme, on the infrastructure of Istanbul Technical University – National Center for High Performance Computing (UHeM). We thank MEXT Technology Center, the programme's operator, and CerebrAI-VortX Neurotechnology and Software Systems for their collaboration on our EEG work.
Run Erk-32B on your own infrastructure
Weights, GGUF quants and every evaluation script are open on Hugging Face. Contact us for enterprise deployment, fine-tuning and integration.