lid-specialists-9plus1

Language-ID specialists for nine languages plus other

Three small models that decide inside confusion groups where a general language identifier is weakest: Malay vs Indonesian, Simplified vs Traditional Chinese, and romanized Hindi vs romanized Urdu vs English. They are stage 2 of a two-stage detector; stage 1 is GlotLID (Apache-2.0), which is not redistributed here.

Trained for a multilingual retrieval application, where misreading the question's language sends the whole answer to the wrong knowledge base.

The three models

The 818 KB head is not an 818 KB model. It needs intfloat/multilingual-e5-small to produce embeddings first, so its real footprint is about 471 MB. Only the fastText model is genuinely small.

Labels (12)

eng_Latn, ind_Latn, zsm_Latn, zho_Hans, zho_Hant, jpn_Jpan, kor_Hang, tam_Taml, hin_Deva, hin_Latn, urd_Latn, other

Every language outside the nine supported ones is other.

Results

FLORES-200 devtest, specialist label space. Measured in this project; the generated table lives in training/RESULTS.md in the source repository.

Read this honestly: the specialists are not better than GlotLID on their own. Used alone, specialist-e5-head scores 0.653 against GlotLID's 0.939. The gain is in the two-stage arrangement, and it is small — 0.942 against 0.939 overall, with the group macro-F1 moving 0.894 to 0.899. What the specialists buy is the confusion groups: the han column goes 0.816 to 0.827, and romanized_hindi 0.992 to 0.994.

The strongest practical result is speed. specialist-fasttext reaches 0.882 macro-F1 at 14,577 sentences/s in 1.9 MB, against GlotLID's 0.939 at 1,538 sentences/s in 1.69 GB.

By input shape

Macro-F1 per slice, plus latency.

Short and code-mixed input is where the specialists earn their place: on single words from Dakshina, GlotLID scores 0.142 and lid.176 scores 0.003, while the fine-tuned model reaches 0.675.

Training data

Not redistributed. The full record is training/data/DATACARD.md in the source repository.

360,000 training rows and 24,000 validation rows, capped at 100,000 lines per label. Cleaning normalizes (NFKC), strips invisible characters, collapses whitespace, and drops lines under two characters and exact duplicates. Synthetic rows: OpenCC Simplified/ Traditional conversions, 1-3 word crops from 10% of natural lines, and code-mixed rows equal to 2% of natural lines.

Contamination: 7,543 training lines were removed for matching a FLORES-200 dev/devtest sentence or a Dakshina test item. The final check found 0 rows overlapping any held-out line across every split.

Training setup

lid-specialist-e5-finetune: learning rate 3e-05, batch size 32, max length 128, 2 epochs maximum, 6% warmup, weight decay 0.01, dropout 0.1, seed 20260915. About 36 minutes on an RTX 5060 Laptop GPU (torch 2.14.0+cu130, bf16). ONNX export agrees with the PyTorch model to 3.8e-06 maximum absolute difference.

Inputs use the e5 convention: prefix query: , max length 128.

Limitations

• OpenCC conversions do not read like natively written Simplified or Traditional Chinese.
• 1-3 word crops are cut from sentences, not collected from real queries.
• Code-mixed rows are two whole sentences joined, labelled by the longer part. Real code-mixing happens inside a sentence.
• The contamination check matches whole lines after normalization, so a held-out sentence embedded inside a longer line is not caught.
• For hin_Latn, urd_Latn, zho_Hans and zho_Hant the test split is a hash split from the same sources as training, not source-disjoint. Those scores are likely optimistic against text from new sources.
• Romanized Hindi and Urdu come from one source (Dakshina: Wikipedia sentences romanized by annotators), so they do not represent romanization as people type it.
• Anything outside the nine languages is only ever other. This is not a general-purpose language identifier; use GlotLID for that.

Licences

Weights are MIT, matching the source repository's code. They are derived from intfloat/multilingual-e5-small (MIT) and trained on the data above; OpenLID's licence is listed as "other" and varies by source corpus, and the underlying data is not redistributed here.

Citation

Comparison figures in the source repository's results come from:

• Kargaran et al. 2023, GlotLID: Language Identification for Low-Resource Languages (arXiv:2310.16248)
• Burchell et al. 2023, An Open Dataset and Model for Language Identification (arXiv:2305.13820)

Those figures come from the papers' own model versions and label mappings. They are context, not a reproduction.