glotlid-onnx

GlotLID v3 - ONNX

ONNX export of cis-lmu/glotlid model_v3.bin, a fastText supervised language identifier that covers 2102 labels in the form iso639-3_Script (for example eng_Latn, por_Latn, glg_Latn).

The licence is Apache-2.0, inherited from GlotLID. Please cite the GlotLID authors if you use this model.

What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

Steps 1 to 3 are string processing. ONNX has no portable operator for fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The split keeps the graph a pure numeric pipeline:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

glotlid_hash.py is the reference implementation of steps 1 to 3. It reproduces fastText's Dictionary::getLine, Dictionary::computeSubwords and Dictionary::hash. Two details are easy to get wrong:

• fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands in the wrong bucket.
• Each line ends with the </s> end-of-sentence token, and that token contributes its own vocabulary row.

Files

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1
nwords=634361  nlabels=2102
input_matrix=(1634361, 256)   # nwords + bucket
output_matrix=(2102, 256)

Tensors

probs is indexed by the order of labels.json.

Usage

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from glotlid_hash import GlotLIDFeaturizer

d = snapshot_download("TigreGotico/glotlid-onnx")
feat = GlotLIDFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/glotlid.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.98...), ...]

Parity with fastText

78 short samples across 60 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Spanish, Basque, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Korean, Hindi, Russian, Ukrainian, Greek, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fasttext_model.predict(text, k=5).

The fp32 probability delta of about 1e-05 is not export error. fastText returns probabilities through a 512-entry logarithm lookup table whose entries carry a +1e-5 term, so its reported values sit about 1e-5 above the true softmax. The ONNX graph returns the exact softmax. This also explains the top-5 order differences: they occur only among labels whose probabilities are below the resolution of that table, where fastText's own ranking is arbitrary.

The int8 graph keeps every top-1 decision but its probability values move by up to 0.018, so use fp32 if you threshold on confidence.

Reproduction

glotlid_hash.py was validated against fastText's own get_subwords() on 2006 tokens (2000 sampled from the vocabulary plus out-of-vocabulary strings in Greek, Devanagari, Han, Latin-extended and Inuktitut syllabics): 0 mismatches in the full feature-id lists.