diacnet-1.0

diacnet-1.0

diacnet-1.0 restores diacritics/accents to text that's been typed or scraped without them, across 10 languages. Fine-tuned from google/byt5-small — character/byte-level rather than word-level, so it handles Yoruba tone marks, Vietnamese combining diacritics, and Polish/Turkish special characters through the same mechanism, no per-language vocabulary needed.

• Single joint model, all 10 languages — a language tag prefix (<yor>, <vie>, etc.) tells the model which diacritic inventory to apply, no separate models or an upstream language-ID step required.
• Median CER of ~0.02 across most languages (see Benchmarks) — near-perfect restoration on well-formed input.
• Fully self-supervised training — no manual annotation. Clean, already- diacritized text is the target; diacritics are deterministically stripped to create the training input.

🗒️ Model Details

Languages: yo vi ig ha pl tr pt es fr it (ISO 639-1; ISO 639-3: yor vie ibo hau pol tur por spa fra ita)

Scoped deliberately to languages where diacritics are lexically meaningful — not applied to the other 15 languages in the source corpus (e.g. Swahili, Zulu, Amharic, Japanese), where diacritic restoration either doesn't apply or isn't the right frame for the script.

🏃 Usage

from transformers import AutoTokenizer, T5ForConditionalGeneration

tok = AutoTokenizer.from_pretrained("olaverse/diacnet-1.0")
model = T5ForConditionalGeneration.from_pretrained("olaverse/diacnet-1.0")

text = "<yor> se eranko naa si gbo o?"
inputs = tok(text, return_tensors="pt")
output_ids = model.generate(**inputs, max_new_tokens=256)
print(tok.decode(output_ids[0], skip_special_tokens=True))
# ṣé ẹranko náà sì gbọ́ ọ?

Prefix input text with the target language tag (<yor>, <vie>, <ibo>, <hau>, <pol>, <tur>, <por>, <spa>, <fra>, <ita>). Works best on single sentences or short passages — see Known Limitations for longer text.

📊 Benchmarks

External evaluation on out-of-distribution test data — not our training distribution. ~1,000 diacritic-bearing test sentences per language: Tatoeba (es/fr/it/pt/pl/tr/vi), MasakhaNEWS (ig/ha), MENYO-20k test split (yo). Sentences were diacritic-stripped to form inputs; the original text is the reference.

Systems: diacnet-1.0 (this model — one joint model, all 10 languages) vs. copy-input (the undiacritized text unchanged — the floor).

WER — word error rate (%, lower is better)

CER — character error rate (%, lower is better)

ChrF (higher is better)

Reading the results honestly

• diacnet-1.0 beats the do-nothing floor on 9 of 10 languages, often by a wide margin: Italian WER 14.2 → 0.6, French 18.2 → 1.5, Vietnamese 86.3 → 21.6.
• Yorùbá remains the hardest language at 29.1 WER — still well above the Romance-language scores. A dedicated Yorùbá-only specialist, Davlan/mt5_base_yoruba_adr (roughly 2× diacnet's size, trained on human-authored text), scored 16.4 WER on an earlier run of this same test set — still ahead of us. Yorùbá tone restoration is largely a semantic disambiguation problem: the same base letters can spell several different words (e.g. ogun war / ògùn medicine / ogún twenty), so closing this gap needs more human-authored Yorùbá training data, not just more parameters.
• Hausa: restoration currently does not work (⚠️). diacnet scores worse than doing nothing (8.3 vs 7.8 WER). With only ~15 Hausa examples in training, the model never learned the hooked-letter inventory (ɓ ɗ ƙ) and largely passes text through unrestored. Do not use diacnet-1.0 for Hausa; fixing this is the top priority for 1.1.
• The earlier ~0.02 median CER figure came from a held-out slice of our own training distribution; the numbers above, on external text, are the ones to trust.

Evaluation code and test sets are open — see DiacBench (olaverse/diacbench) — so these numbers are reproducible and other models can be compared on the same footing.

🔜 Toward diacnet-1.1

1. Hausa — train on adequate Hausa data (Wura, MasakhaNEWS); target: clearly beat the copy-input floor.
2. Yorùbá — add human-authored training text (e.g. MENYO-20k train split) to close the gap to the monolingual specialist.
3. Turkish/Vietnamese — remaining errors concentrate in lexically ambiguous cases; larger-capacity variants under evaluation.

Training data & licensing

Fine-tuned from google/byt5-small (Apache-2.0) on olaverse/qg-passages-multi (Apache-2.0), split into sentences and paired with a diacritic-stripped copy of each sentence as the self-supervised training input. Released under Apache-2.0.

Citation

@misc{diacnet-1.0,
  title  = {diacnet-1.0},
  author = {Olaverse},
  year   = {2026},
  url    = {https://huggingface.co/olaverse/diacnet-1.0}
}