yobyt5-restoration

Yo-ByT5: Byte-Level Diacritic Restoration for Yorùbá

Yo-ByT5 is a fine-tune of google/byt5-small that restores Yorùbá diacritics (tone marks: acute ◌́, grave ◌̀, mid unmarked; underdots: ẹ, ọ, ṣ) to undiacritized text.

• Developer: Ahmad Samuel Gali (lazymonster)
• Paper: Yo-ByT5: Efficient and High-Fidelity Diacritic Restoration for Yorùbá
• Language: Yorùbá (yo)
• Task: Automatic Diacritic Restoration (ADR)
• License: Apache 2.0

---

Quickstart

import unicodedata
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "lazymonster/yobyt5-restoration"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "Eko ni kokoro aseyori."
inputs = tokenizer(text, return_tensors="pt", max_length=1024, truncation=True)
outputs = model.generate(inputs["input_ids"], max_length=1024, num_beams=1)

predicted = tokenizer.decode(outputs[0], skip_special_tokens=True)
predicted = unicodedata.normalize("NFC", predicted)
print(predicted)
# Ẹ̀kọ́ ni kọ́kọ́rọ́ àṣeyọrí.

---

Benchmark Results

Evaluated on the YAD Test Benchmark (3,330 sentences) using base-word Needleman-Wunsch sequence alignment and Unicode NFC normalization.

The Corrected Reference repairs 1,415 non-standard underdot codepoints (U+0329 to standard U+0323) in the raw YAD release.

Greedy decoding uses num_beams=1. Beam search uses num_beams=5.

---

Training Data

---

Training Procedure

• Architecture: google/byt5-small (299.6M parameters)
• Compute: Google Cloud TPU v6e-8
• Optimizer: AdamW, linear decay, weight decay 0.01, gradient clip norm 0.5
• Batch Size: Per-device batch 4, gradient accumulation 2 (effective global batch size 64)
• Phase 1: Learning rate 2e-4, 300 warmup steps, 4 epochs
• Phase 2: Learning rate 1e-4, 0 warmup steps, 4 epochs (released checkpoint)

---

Recommendations

1. Unicode normalization: Always normalize input and generated text to NFC before scoring or processing.
2. Chunking: Chunk inputs longer than 1024 bytes at sentence or clause boundaries.
3. Decoding: Greedy decoding is fast and accurate. Beam search (num_beams=5) offers slight quality gains on edit distance metrics.

---

Citation & License

License

• Model weights: Apache 2.0
• Training data: Subject to original licenses (MENYO-20k: CC BY-NC 4.0; Biblica: CC BY-SA)

Citation

@misc{gali2026yobyt5efficienthighfidelitydiacritic,
      title={Yo-ByT5: Efficient and High-Fidelity Diacritic Restoration for Yor\`ub\'a}, 
      author={Ahmad Samuel Gali and Shamsuddeen Hassan Muhammad},
      year={2026},
      eprint={2610.01634},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.01634}, 
}

Acknowledgments

Trained as a member of the HausaNLP Research Group with compute support from the Google TPU Research Cloud (TRC) program.