openlid-v2-onnx

OpenLID-v2 - ONNX

ONNX export of laurievb/OpenLID-v2 model.bin, an improved fastText supervised language identifier covering 200 language varieties in iso639-3_Script label form (for example eng_Latn, por_Latn, glg_Latn), built on the OpenLID-v2 dataset, an updated version of Burchell et al., An Open Dataset and Model for Language Identification (ACL 2023).

The licence is GPL-3.0, inherited unchanged from the original model.

[!NOTE] The upstream model card recommends normalising/cleaning text with openlid_normer.clean_line before classification for best results. This export does not vendor that normalizer - it hashes raw text exactly like TigreGotico/openlid-onnx and TigreGotico/glotlid-onnx do, and parity was measured against fastText's own predict() on raw (uncleaned) text, so the two stay comparable. Apply the same cleaning step yourself before calling the featurizer if you want to match the upstream-recommended pipeline.

What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

Steps 1 to 3 are string processing. ONNX has no portable operator for fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The split keeps the graph a pure numeric pipeline:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

openlid_v2_hash.py is the reference implementation of steps 1 to 3, ported from the same recipe used for TigreGotico/glotlid-onnx. Two details are easy to get wrong:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands in the wrong bucket.
• Each line ends with the </s> end-of-sentence token, and that token contributes its own vocabulary row.

This model uses plain softmax loss (not hierarchical softmax), so it does not need the Huffman-tree combination step that TigreGotico/lid176-onnx (fastText's classic lid.176.bin, trained with loss=hs) requires.

Files

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
nwords=185286  nlabels=200
input_matrix=(1185286, 256)   # nwords + bucket
output_matrix=(200, 256)

Tensors

probs is indexed by the order of labels.json.

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from openlid_v2_hash import FastTextFeaturizer

d = snapshot_download("TigreGotico/openlid-v2-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/openlid-v2.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.9...), ...]

Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek, Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fastText's own f.predict(text + "\n", 1, 0.0, "strict") (the pybind entry point - fasttext-wheel's Python .predict() wrapper is broken under numpy>=2), on raw uncleaned text (see the note above about the upstream normalizer).

Citation

@inproceedings{burchell-etal-2023-open,
    title = "An Open Dataset and Model for Language Identification",
    author = "Burchell, Laurie and Birch, Alexandra and Bogoychev, Nikolay and Heafield, Kenneth",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    year = "2023",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.75",
}