lid176-onnx

fastText lid.176 - ONNX

ONNX export of Facebook/Meta's classic fastText language identification model, lid.176.bin (also mirrored at julien-c/fasttext-language-id). It covers 176 languages with two-letter/three-letter fastText label codes (for example __label__en, __label__pt, __label__zh).

The licence is CC-BY-SA-3.0, inherited unchanged from the original Facebook release. This is a re-export, not a re-training - please keep attribution and licence terms intact if you redistribute it.

What the graph does and does not do - and a wrinkle this model has

fastText inference has five steps:

Steps 1-3 are string processing with no portable ONNX op for fastText's FNV-1a byte hash, so they stay in Python (lid176_hash.py), same as every other model in this hashing family.

Step 5 is where lid.176.bin differs from GlotLID and OpenLID: this model was trained with fastText's hierarchical softmax (loss=hs), not a flat softmax. Under hierarchical softmax the output matrix's rows are binary classifiers for internal nodes of a Huffman tree built over the label frequencies - a label's probability is the product of the sigmoid (or 1-sigmoid) values along the root-to-leaf path, not a single row of the output matrix. A plain MatMul -> Softmax graph (the GlotLID/OpenLID recipe) gives 0% agreement with fastText on this model - it was tried and verified wrong before switching to the approach below.

The ONNX graph therefore ends in Sigmoid over every Huffman node, and a small Python step (hs_tree.py + HSCombiner in lid176_hash.py) walks each label's path to combine node probabilities into per-label ones:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0)
    -> MatMul(output_matrix) -> Sigmoid -> node_probs

node_probs = onnx_session.run(...)          # sigmoid per Huffman node
probs = HSCombiner.from_file("hs_tree.json")(node_probs)   # per label

The Huffman tree is not stored in fastText's .bin file - it is rebuilt deterministically from the label frequency counts every time the model loads. hs_tree.py reproduces that construction (HierarchicalSoftmaxLoss:: buildTree in fastText's Loss.cc) exactly, and hs_tree.json bakes the resulting per-label paths/codes in at export time so no fastText C++ code is needed at inference.

Two details that are easy to get wrong, both already solved here:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Missing this mis-hashes every non-ASCII n-gram.
• Each line ends with the </s> end-of-sentence token, which has its own vocabulary row and is a strong learned prior - always append it (the Python featurizer does this for you).

Files

Model arguments

dim=16  minn=2  maxn=4  bucket=2000000  wordNgrams=1  loss=hs
nwords=40010  nlabels=176
input_matrix=(2040010, 16)   # nwords + bucket
output_matrix=(176, 16)      # rows 0..174 are Huffman-node classifiers

Tensors

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from lid176_hash import FastTextFeaturizer, HSCombiner

d = snapshot_download("TigreGotico/lid176-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
combine = HSCombiner.from_file(f"{d}/hs_tree.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/lid176.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    node_probs = sess.run(None, {"input_ids": feat(text)})[0]
    probs = combine(node_probs)
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("The weather is very nice today in London."))
# [('__label__en', 0.99...), ...]

Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek, Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fastText's own f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's Python .predict() wrapper is broken under numpy>=2).

The two int8 mismatches are close calls between historically related languages/creoles (pms/ht, it/pt) that swap only after 8-bit weight rounding; fp32 gets both right.

Citation

If you use this model, please cite the original fastText/lid.176 authors:

@article{joulin2016fasttext,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}
@article{joulin2016fasttext2,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J\'egou, H\'erve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}