lid-neural-5.1

lid-neural-5.1

A compact language identifier for the four main Nigerian languages — Hausa (hau), Yoruba (yor), Igbo (ibo), and Nigerian Pidgin (pcm) — built as a classification head on olaverse/mist-encoder-base-ng.

It is trained on sentence-level text, so it works on short inputs (tweets, queries, single sentences), not just long documents. At ~31M parameters it runs comfortably on CPU / edge.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("olaverse/lid-neural-5.1")
model = AutoModelForSequenceClassification.from_pretrained("olaverse/lid-neural-5.1")

text = "Ina kwana?"
inputs = tok(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]
pred = model.config.id2label[int(probs.argmax())]
print(pred, float(probs.max()))   # -> Hausa 0.99

Labels: Hausa, Yoruba, Igbo, Nigerian Pidgin (see id2label in the config).

Evaluation

Held-out test set (sentence-level), balanced across the four languages.

Confusion matrix (rows = true, cols = predicted):

Hausa, Yoruba, and Igbo are separated near-perfectly (1–4 cross-errors each). Almost all residual confusion involves Nigerian Pidgin, which is English-lexified and shares surface vocabulary with the others — so it is the hardest class and the main source of error.

Training data

Sentence-level text drawn from our own monolingual corpora — each language's text is its own label, so no separate labelled dataset was needed. Sources: FineWeb-2 (ODC-By) for ha/yo/ig/pcm and the Nigerian Pidgin ASR corpus (CC-BY-4.0) for additional Pidgin. All attribution-only, consistent with the Apache-2.0 release.

Limitations

• No English / "other" class. The model always predicts one of the four Nigerian languages. English (or any out-of-set language) will be confidently mislabelled — most often as Nigerian Pidgin, since that is where the decision boundary sits. If your inputs may contain English or other languages, this model is not yet suitable on its own. (An English class is the planned v5.2 improvement.)
• Pidgin is the weakest class (see confusion matrix), by nature rather than by defect.
• Trained and evaluated on written text; performance on heavily code-mixed or non-standard orthography may be lower.

License

Apache-2.0. Training data is attribution-only (ODC-By / CC-BY-4.0); please retain attribution to the upstream corpora.