megazeka-byt5-tr-spellfix

Megazeka · Turkish Spelling Correction (ByT5-small + LoRA)

A LoRA adapter for google/byt5-small that corrects Turkish spelling, typos, punctuation, capitalization, spacing, common keyboard errors and chat-style colloquial spellings (gidicem → gideceğim, bilmiyom → bilmiyorum, burda → burada). Trained locally on a single RTX 4060.

This is a small educational experiment, not a production-grade spell checker. Read the limitations before using it for anything that matters.

• Code, offline desktop app and full technical report: github.com/pandakingpunc/megazeka
• Training data: pandakingpunc/megazeka-tr-spellfix-pairs

What's new in v2

The first release (v1) did not correct colloquial forms at all — gidicem, buluşcaz, gidicez came out unchanged — because its training data was almost entirely literary third-person sentences (gideceğim appeared 8 times in 32,000 pairs). v2 changes the data, not the architecture:

• 4,904 everyday first/second-person pairs generated from project-authored templates (50 verbs and a small conjugator) plus ~250 hand-written short sentences, all CC0-1.0.
• Rule-based colloquial noise on suffixes, so it applies to any verb: -eceğim → -icem/-cem, -acağız → -ıcaz/-caz, -yorum → -yom, -yor → -yo, consonant hardening (gideceğim → gitcem), plus word rules (burada → burda, bir şey → bişey, herkes → herkez, yalnız → yanlız, …). Colloquial operations in the training split went from 1,001 to 8,178.
• 1,153 two-sentence training chunks, so the model sees more than one sentence per input like it does when a user pastes a paragraph.
• Longer training: 4,000 steps instead of 1,800.
• Two-layer output guard in the reference inference code (see Usage): a looser chunk filter plus a new word-level lock that blocks synonym swaps and invented words.

On the same 160 held-out Common Voice target sentences (noised differently in each version, so the pairs are not identical), exact match went from 60.6% (v1) to 70.3% (v2). The v1 adapter is still available with revision="v1".

Why byte-level

Turkish spelling errors are overwhelmingly diacritic errors: ı/i, İ/I, ş/s, ğ/g, ç/c, ö/o, ü/u. Subword tokenizers fragment unpredictably when those characters are corrupted, so a misspelled word and its correction can land in completely unrelated token space. ByT5 operates on raw UTF-8 bytes and has no vocabulary to fall out of, which makes it a natural fit for this task. The cost is sequence length — a Turkish character is often two bytes.

Results

Held-out test set: 1,128 pairs derived from 282 target sentences that appear in no other split — 640 pairs from 160 Common Voice sentences and 488 pairs from 122 everyday sentences. Copy input is the trivial do-nothing baseline. Base ByT5 is untrained google/byt5-small.

By subset (guard on):

Untrained ByT5 is unusable here — its raw CER is 4.04 and it rewrites every input, so the output guard falls back to the input almost everywhere. That is why its protected column is close to copy-input.

Unfiltered is the raw generation with the output guard (see below) disabled. The guard costs 0.4 points of exact match and buys protection against word substitutions and large rewrites; both are reported because which one you want depends on your use case.

Overcorrection is measured on the 282 test examples where input already equals target (5 of them were changed).

Edit precision/recall/F1 compares sets of (original position, operation, character) Levenshtein edits; with the guard on, precision is 0.926 and recall 0.871. This is a project-defined metric and is not comparable to standard GEC M2/ERRANT F0.5.

Usage

from transformers import AutoTokenizer, T5ForConditionalGeneration
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("google/byt5-small")
base = T5ForConditionalGeneration.from_pretrained("google/byt5-small")
model = PeftModel.from_pretrained(base, "pandakingpunc/megazeka-byt5-tr-spellfix").eval()

text = "bugün okula gidicem ama hava cok kötü galiba"
ids = tok(text, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=224, num_beams=1, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
# Bugün okula gideceğim ama hava çok kötü galiba.

For the previous release: PeftModel.from_pretrained(base, "pandakingpunc/megazeka-byt5-tr-spellfix", revision="v1").

Things the repo's own inference code (src/megazeka/inference.py) does that the snippet above does not:

1. Normalization. NFC Unicode and line-ending normalization only.
2. Chunking. Input is split into pieces of at most 176 UTF-8 bytes on whitespace boundaries; line breaks and blank paragraphs are preserved. No context crosses chunk borders.
3. Chunk filter. A chunk's generation is discarded and the original kept if the character edit count exceeds the input length (ratio > 1.0), the output is shorter than 65% of the input, the output is empty, or no EOS token was produced. (v1 used a 45% edit-ratio threshold, which rejected most correct heavy fixes: edit F1 0.71 → 0.89 when relaxed, with overcorrection unchanged.)
4. Word lock. Words are aligned ignoring case, Turkish diacritics and punctuation. If a word group was replaced by one with no letter similarity (normalized Levenshtein distance > 0.6), a word was deleted, or a new word appeared with no counterpart, only that group is restored to the original. So gidicem → gideceğim passes, okula → eğitim kurumuna does not.

The guard does not guarantee meaning preservation, and it also blocks some correct fixes. Generation is deterministic greedy decoding (num_beams=1, do_sample=False).

What it actually does (and doesn't)

Real outputs from this adapter on hand-picked sentences outside the training data (raw generation and guarded output are identical for all of these):

17 of these 23 hand-picked examples come out exactly right (73.9%); 1 of the 9 already-correct ones was changed (the Işık proper noun). This is a sanity check, not a success rate.

Training

Best checkpoint selected by target-token-weighted loss on 640 validation pairs: step 4,000, loss 0.0376 (v1: 0.0554).

Data

38,057 training pairs from 9,226 distinct targets:

• 32,000 from 8,000 sentences of the Common Voice Turkish Sentence Collector (CC0-1.0, pinned commit),
• 4,904 from project-authored everyday sentences (training/everyday.py, CC0-1.0),
• 1,153 two-sentence chunks built by joining two pairs of the same difficulty (≤ 170 bytes).

Each target yields four variants — clean, easy, medium, hard — targeting 0 / 1 / 2 / 4 applied noise operations; 25% of examples are deliberately left clean so the model learns not to touch correct text. For easy and medium everyday pairs the first operation is always a colloquial transformation.

Noise operations include letter deletion/repetition/transposition, Turkish Q-keyboard neighbours, diacritic loss, case errors, punctuation, merged/split words, extra whitespace, de/da and ki clitic merging, question-suffix and apostrophe loss, and suffix-based colloquial spellings. Every applied operation is recorded with its before/after text in the dataset.

Splitting happens on canonical target form before noise is applied, so no variant of the same sentence crosses splits. This is not a claim that semantically similar sentences are separated, nor that these sentences are absent from ByT5's pretraining corpus — that overlap is unknown.

Limitations

• Synthetic noise does not cover the full variety of real user errors. Expect weaker performance on genuine user text than these numbers suggest.
• The everyday subset comes from 50 verbs and a limited set of complements; it does not represent real chat language in full, and the 84% everyday score is on sentences of the same template family.
• Colloquial forms are sometimes expanded imperfectly: consonant softening can be skipped (gitcez → giteceğiz instead of gideceğiz) and contractions like napcam are not fully expanded.
• Heavy abbreviations (tşk, inş) can be held back by the word lock and left unchanged.
• Weak on rare and proper nouns (Işık → İşık), code-switched text, context-dependent de/da/ki, missing apostrophes on proper nouns, and stylistic punctuation choices.
• No context is carried across 176-byte chunks.
• Meaning-altering edits can still occur despite clean-text training and the output guard.
• Token probabilities exposed by the desktop app are not calibrated confidence scores and should not be read as per-correction certainty.
• Trained and evaluated on sentence-length text (18–160 UTF-8 bytes, 3–24 words, plus two-sentence chunks). Behaviour on long documents, lists, code or mixed-language text is untested.

Versions

Licensing

• This adapter: Apache-2.0, matching the google/byt5-small base model.
• Project source code: MIT (see the GitHub repository).
• Training sentences: CC0-1.0, from Common Voice and project-authored everyday sentences. No Common Voice program code is included here.

Citation

@software{megazeka2026,
  title   = {Megazeka: Turkish Spelling Correction with ByT5-small and LoRA},
  author  = {pandakingpunc},
  year    = {2026},
  version = {2},
  url     = {https://github.com/pandakingpunc/megazeka}
}