openlid-onnx

OpenLID - ONNX

ONNX export of laurievb/OpenLID model.bin, a fastText supervised language identifier covering 201 languages in iso639-3_Script label form (for example eng_Latn, por_Latn, glg_Latn), from Burchell et al., An Open Dataset and Model for Language Identification (ACL 2023).

The licence is GPL-3.0, inherited unchanged from the original model. There is a newer OpenLID-v2 covering slightly fewer but revised language varieties - use that one for new projects unless you need parity with the original OpenLID paper.

What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

Steps 1 to 3 are string processing. ONNX has no portable operator for fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The split keeps the graph a pure numeric pipeline:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

openlid_hash.py is the reference implementation of steps 1 to 3, ported from the same recipe used for TigreGotico/glotlid-onnx. Two details are easy to get wrong:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands in the wrong bucket.
• Each line ends with the </s> end-of-sentence token, and that token contributes its own vocabulary row.

This model uses plain softmax loss (not hierarchical softmax), so it does not need the Huffman-tree combination step that TigreGotico/lid176-onnx (fastText's classic lid.176.bin, trained with loss=hs) requires.

Files

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
nwords=201110  nlabels=201
input_matrix=(1201110, 256)   # nwords + bucket
output_matrix=(201, 256)

Tensors

probs is indexed by the order of labels.json.

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from openlid_hash import FastTextFeaturizer

d = snapshot_download("TigreGotico/openlid-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/openlid.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.9...), ...]

Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek, Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fastText's own f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's Python .predict() wrapper is broken under numpy>=2).

Citation

@inproceedings{burchell-etal-2023-open,
    title = "An Open Dataset and Model for Language Identification",
    author = "Burchell, Laurie and Birch, Alexandra and Bogoychev, Nikolay and Heafield, Kenneth",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    year = "2023",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.75",
}