fasttext-lid-176-sinhala-pali-sanskrit

fastText LID-176: Sinhala-Script Disambiguation (Pali, Sanskrit, Sinhala)

This repository contains the fine-tuned specialist fastText model for Sinhala-script Language Identification, developed as part of the paper:

"A Benchmark for Sinhala-Script Language Identification: Disambiguating Sinhala, Pali, and Sanskrit in Closed and Open World Contexts"

1. Problem Overview

Planetary-scale LangID tools (including stock fastText LID-176) assign all text written in the Sinhala script to a monolithic si label. When presented with historical or canonical Buddhist and scholarly literature written in Sinhala script, stock models fail completely:

• Pali in Sinhala script (pli_Sinh): 0.0% F1 (Stock zero-shot)
• Sanskrit in Sinhala script (san_Sinh): 0.0% F1 (Stock zero-shot)

2. Two-Stage Specialist Routing Architecture

To eliminate script ambiguity while guaranteeing mathematical 0.0% degradation on all 176 global background languages, we deploy this model in a Two-Stage Specialist Routing Pipeline:

1. Stage 1 (Global Router): Stock fastText LID-176 (lid.176.bin) classifies incoming text.
   • If the prediction is NOT Sinhala (!= 'si'), the global label (English, Tamil, Hindi, Devanagari Sanskrit, French, etc.) is emitted directly.
2. Stage 2 (Specialist Model - this repository): If Stage 1 detects Sinhala script (== 'si'), the input is routed to this specialist model (model.bin), which performs fine-grained 3-way discrimination:
   • __label__sinhala (Sinh-Sinh)
   • __label__pali (Pali-Sinh)
   • __label__sanskrit (San-Sinh)

3. Evaluation on 11-Language Hybrid Benchmarks

(Note: Background languages such as English, Tamil, Arabic, Bengali, French, and German retain their pristine stock fastText performance).

4. Quickstart / Usage

import fasttext
from huggingface_hub import hf_hub_download

# 1. Download Stage 1 (Stock FastText) and Stage 2 (Specialist)
stage1_path = hf_hub_download(repo_id="facebook/fasttext-language-identification", filename="model.bin")
stage2_path = hf_hub_download(repo_id="script-langid/fasttext-lid-176-sinhala-pali-sanskrit", filename="model.bin")

stage1_model = fasttext.load_model(stage1_path)
stage2_model = fasttext.load_model(stage2_path)

def identify_language(text: str):
    # Stage 1: Global screening
    pred1, prob1 = stage1_model.predict(text.replace("\n", " "))
    label1 = pred1[0].replace("__label__", "")
    
    # If Sinhala script detected by Stage 1, route to Stage 2
    if label1 in ["si", "sin", "sin_Sinh"]:
        pred2, prob2 = stage2_model.predict(text.replace("\n", " "))
        target_label = pred2[0].replace("__label__", "")
        return {
            "language": f"{target_label.capitalize()}-Sinh",
            "stage": 2,
            "confidence": float(prob2[0])
        }
    else:
        return {
            "language": label1,
            "stage": 1,
            "confidence": float(prob1[0])
        }

# Example: Pali Buddhist Verse in Sinhala Script
sample_pali = "මනොපුබ්බඞ්ගමා ධම්මා මනොසෙට්ඨා මනොමයා"
print(identify_language(sample_pali))
# Output: {'language': 'Pali-Sinh', 'stage': 2, 'confidence': 0.99...}

5. Citation

@inproceedings{sinhala_script_langid_2026,
  title={A Benchmark for Sinhala-Script Language Identification: Disambiguating Sinhala, Pali, and Sanskrit in Closed and Open World Contexts},
  author={Anonymous},
  booktitle={Proceedings of the Association for Computational Linguistics (ACL)},
  year={2026}
}