yoruba-codeswitch-diacritics-long-context

Yorùbá-English Code-Switching Diacritic Restoration (Long-Context)

This model restores missing tone marks and underdots (diacritics) in Yorùbá text within code-switched Yorùbá-English contexts. It is explicitly designed and optimized to handle long-context sequences up to 2048 tokens, enabling the accurate restoration of full paragraphs, documents, and extended conversational text without the truncation or coherence loss typical of standard 512-token models.

📈 Training Dynamics

Figure 1: Phase 3 training and validation loss curves over 5,398 steps. The model achieved a best validation loss of 0.03448 at step 5398, demonstrating stable convergence and successful adaptation to the 2048-token context window without catastrophic forgetting.

---

📋 Model Details

• Model Architecture: ByT5-small (Byte-level T5)
• Base Model: google/byt5-small
• Total Parameters: ~300 million
• Maximum Context Window: 2048 tokens
• Supported Languages: Yorùbá (primary), English (code-switching support)
• Primary Task: Sequence-to-sequence automatic diacritic restoration (ADR)
• License: MIT

---

🏗️ Training Methodology

Progressive Context Expansion (Curriculum Fine-Tuning)

To enable the stable training of a 2048-token context window on consumer-grade hardware (NVIDIA RTX 3070 Laptop, 8GB VRAM), this model was trained using a rigorous 3-Phase Curriculum Fine-Tuning strategy. This methodology prevents catastrophic forgetting of foundational orthographic rules while gradually adapting the model's attention mechanisms to longer, more complex dependencies.

Phase 3 Training Configuration

The final phase was trained under the following strict hyperparameter configuration to ensure maximum stability and performance:

• Hardware: NVIDIA RTX 3070 Laptop GPU (8GB VRAM)
• Precision: Mixed Precision (BF16/FP16) for memory efficiency
• Batch Size: 1 (with gradient accumulation to simulate larger effective batch sizes)
• Learning Rate: 2e-5
• Learning Rate Scheduler: Linear decay with warmup
• Warmup Steps: 500
• Optimizer: AdamW
• Weight Decay: 0.01
• Maximum Training Steps: 5,398 steps (equivalent to 2 full epochs over the 60,000 sample dataset)
• Evaluation Strategy: Evaluated every 500 steps
• Save Strategy: Checkpoint saved every 1,000 steps
• Best Checkpoint Selected: Step 5,398 (Validation Loss: 0.03448)
• Total Training Time: ~11.8 hours
• Total Floating Point Operations (FLOPs): 1.41 × 10¹⁷

Strict Dataset Filtering Protocol

A critical methodological requirement for long-context evaluation is preventing silent truncation. If a model is trained on a maximum of 2048 tokens, evaluating it on sequences longer than 2048 tokens will result in the tokenizer silently dropping the end of the sequence, artificially inflating error rates (CER/WER/DER) due to forced deletion errors.

To ensure scientific rigor and accurate metric reporting, the Phase 3 test set was strictly filtered based on true token length:

• Original Unfiltered Test Set: 6,391 samples
• Filtered Test Set (≤ 2048 tokens): 5,388 samples (84.3% coverage)
• Excluded Samples: 1,003 samples (15.7%) that exceeded the 2048-token limit after tokenization.

All reported evaluation metrics below are computed exclusively on this filtered 5,388-sample test set to guarantee that the model was evaluated fairly within its designed operational limits.

---

📊 Evaluation Results

Evaluation Protocol

All evaluations were conducted on the filtered test set (5,388 samples ≤ 2048 tokens). The following orthogonal metrics were computed to provide a comprehensive view of model performance:

• CER (Character Error Rate): Levenshtein edit distance at the character level. Measures overall typographical accuracy.
• WER (Word Error Rate): Levenshtein edit distance at the word level. Measures usability and grammatical integrity.
• DER (Diacritic Error Rate): Custom metric measuring the error rate specifically on characters that require diacritics (tone marks and underdots), ignoring plain ASCII characters.
• WDER (Word Diacritic Error Rate): The percentage of diacritic-containing words in the reference that have ≥1 diacritic error in the prediction.
• chrF (Character n-gram F-score): Measures character-level n-gram overlap. Highly sensitive to diacritic restoration quality, as it rewards partial matches better than exact word matching.
• EM (Exact Match): The percentage of sequences where the prediction matches the reference 100% identically.

Custom Long-Context Evaluation (5,388 samples)

Note: Beam Search (num_beams=4) evaluation is currently in progress. Final metrics will be updated in this table upon completion.

---

🚀 Usage

Method 1: Simple Transformers Pipeline

from transformers import pipeline

# Load the model pipeline
diacritizer = pipeline(
    "text2text-generation", 
    model="Emeritus-21/yoruba-codeswitch-diacritics-long-context",
    device=0  # Set to 0 for GPU acceleration, -1 for CPU
)

# Input code-switched text without diacritics
text = "mo ri oko ni ilu Eko because I was driving yesterday"

# Generate restored text (Beam search recommended for maximum accuracy)
result = diacritizer(
    text, 
    max_new_tokens=2048, 
    num_beams=4,  
    early_stopping=True
)

print(result[0]['generated_text'])
# Expected output: "mo rí okò ní ìlú Èkó because I was driving yesterday"