openlid-v2-sinhala-pali-sanskrit

OpenLID-v2: Sinhala-Script Disambiguation (Pali, Sanskrit, Sinhala)

This repository contains the fine-tuned OpenLID-v2 model for Sinhala-script Language Identification, developed as part of the paper:

"A Benchmark for Sinhala-Script Language Identification: Disambiguating Sinhala, Pali, and Sanskrit in Closed and Open World Contexts"

1. Problem Overview

Stock OpenLID-v2 (laurievb/OpenLID-v2) supports 200+ languages, but designates only a single generic label (sin_Sinh) for texts in the Sinhala script. When evaluated zero-shot on corpora containing Pali and Sanskrit written in Sinhala script:

• Pali in Sinhala script (pli_Sinh): 0.0% F1 (Stock zero-shot)
• Sanskrit in Sinhala script (san_Sinh): 0.0% F1 (Stock zero-shot)
• All target texts are indiscriminately absorbed into sin_Sinh.

2. Adaptation & Architecture

This model extends the OpenLID-v2 representation space via supervised transfer learning on a group-stratified corpus of 60,285 target sentences across Sinhala, Pali, and Sanskrit in Sinhala script.

When deployed in a Two-Stage Specialist Pipeline:

• Non-Sinhala scripts are handled directly by the global 200+ language heads.
• Sinhala-script inputs (sin_Sinh) are classified into granular classes:
  • __label__sinhala (Sinh-Sinh)
  • __label__pali (Pali-Sinh)
  • __label__sanskrit (San-Sinh)

3. Evaluation on 11-Language Hybrid Benchmarks

Note on the "Dialect Continuum Problem" & Devanagari Sanskrit

OpenLID-v2 attains lower single-label Macro-F1 scores on South Asian regional classes due to its fine-grained regional dialect heads. When presented with classical Devanagari Sanskrit text, OpenLID-v2 fragments probability mass across genetically descended dialect heads, predicting:

• 68.1% as Bhojpuri (bho_Deva)
• 15.1% as Maithili (mai_Deva)
• 11.0% as Awadhi (awa_Deva) While linguistically sensible across the Indo-Aryan dialect continuum, this reduces single-label benchmark F1 for Devanagari Sanskrit (0.0350). Target Sinhala-script discrimination remains unaffected at $>0.97$ F1.

4. Quickstart / Usage

import fasttext
from huggingface_hub import hf_hub_download

# Download model from Hugging Face
model_path = hf_hub_download(repo_id="script-langid/openlid-v2-sinhala-pali-sanskrit", filename="model.bin")
model = fasttext.load_model(model_path)

def predict_lang(text: str):
    clean_text = text.replace("\n", " ").strip()
    labels, probs = model.predict(clean_text, k=1)
    label = labels[0].replace("__label__", "")
    return {"label": label, "confidence": float(probs[0])}

sample_text = "නමො තස්ස භගවතො අරහතො සම්මාසම්බුද්ධස්ස"
print(predict_lang(sample_text))
# Output: {'label': 'pali', 'confidence': 0.99...}

5. Citation

@inproceedings{sinhala_script_langid_2026,
  title={A Benchmark for Sinhala-Script Language Identification: Disambiguating Sinhala, Pali, and Sanskrit in Closed and Open World Contexts},
  author={Anonymous},
  booktitle={Proceedings of the Association for Computational Linguistics (ACL)},
  year={2026}
}