SindhiFormer-Validation

SindhiFormer — MBAA Architecture Validation

Morpheme-Boundary-Aware Attention (MBAA) for Arabic-Script Low-Resource Language Models

This repository contains the complete validation evidence for SindhiFormer, the first Transformer architecture designed specifically for Sindhi with a novel morpheme-aware attention mechanism.

Key Results

All 3 random seeds beat the standard baseline — the improvement is consistent, not a lucky seed artifact.

Architecture Validated

SindhiFormer (Validation Scale)
├── 6 Layers / 6 Attention Heads / 384 Hidden Dim (~23M params)
├── RoPE (Rotary Position Embeddings)
├── SwiGLU Activation
├── Pre-RMSNorm
├── Weight Tying
└── MBAA: 3 of 6 heads receive morpheme boundary bias (learnable)

What is MBAA?

Standard attention treats all token pairs equally. MBAA adds a learnable negative bias at morpheme boundaries, telling selected attention heads: "tokens within the same word should attend to each other more strongly."

The bias is one scalar per layer — total parameter overhead: 6 floats for the entire model. Yet it produces consistent perplexity improvement because it provides a morphological inductive bias that would otherwise require millions of extra tokens to learn statistically.

MBAA biases learned per-layer patterns:

• Layer 0 (shallowest): bias ≈ -1.00 (barely moved — low-level processing doesn't need morpheme signal)
• Layers 1-3 (middle): bias ≈ -0.70 to -0.82 (strongest movement — morphological composition happens here)
• Layer 5 (deepest): bias ≈ -0.77 (moderate — uses signal for higher-level decisions)

Validation Experiments

Three progressive validation runs were conducted:

Production Architecture (Planned)

Based on validation results and scaling law analysis:

SindhiFormer-62M (Production)
├── 16 Layers / 8Q+2KV Heads (GQA) / 512 Hidden Dim
├── SwiGLU (d_ff=1408) / RoPE / Pre-RMSNorm
├── MBAA on 3 of 8 query heads
├── 16,384 vocabulary (morpheme-aware BPE, 1.06 fertility)
└── Training: 4 epochs on 505M tokens (~2B effective), TPU v3-8

Files in This Repository

Related Resources

Applicability to Other Languages

This architecture is directly transferable to other Arabic-script, morphologically-rich, low-resource languages:

• Urdu (same script family, similar morphology)
• Pashto (Arabic script, agglutinative)
• Kurdish Sorani (Arabic script)
• Balochi (Arabic script, low-resource)

Replace SindhiNLTK's morpheme rules with the target language's morphological analyzer. The rest of the architecture requires no modification.

Author

Aakash Meghwar — Computational Linguist & Independent Researcher
HuggingFace · GitHub

Citation

@misc{meghwar2026sindhiformer,
  author    = {Aakash Meghwar},
  title     = {SindhiFormer: Morpheme-Boundary-Aware Attention for Arabic-Script Language Models},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/aakashMeghwar01/SindhiFormer-Validation}
}