Dihya-5M

Dihya-5M

A 5.19M-parameter byte-level Conv-Transformer that separates six Latin-script Amazigh varieties from each other and from an explicit NOT_AMAZIGH rejection class: Kabyle (kab_Latn), Tashelhit (shi_Latn), Tarifit (rif_Latn), Tamasheq (taq_Latn), Central Atlas Tamazight (tzm_Latn) and Shawiya (shy_Latn).

On a balanced, source-stratified held-out set of 1,050 sentences it reaches 85.52% accuracy and 0.8562 macro-F1 over the seven classes. Scored on the six Berber classes alone — the label set the public identifiers can be asked about — it reaches 84.44% accuracy, against 41.78% for GlotLID and 28.33% for NLLB's lid218e measured in the same pass over the same rows.

Three of the six — Tarifit, Central Atlas Tamazight and Shawiya — have no label in either public identifier, so this is the first system that can name them at all.

It ships with a split-conformal quantile, so it can return a set of labels rather than one, for pipelines where a silent misclassification costs more than a widened answer.

The task

These are not typologically distant languages. The six varieties share much of a core lexicon — aman (water), argaz (man), awal (word) — and one Latin orthography. The evidence that separates them is affixal, phonotactic and clitic, and a short sentence of shared vocabulary often carries none of it.

That matters beyond the benchmark. NLLB's own identifier mined 90.1% of this project's parallel corpus, and it labels the siblings Kabyle: 91.3% of Shawiya, 80.7% of Tarifit, 79.3% of Tashelhit and 73.3% of Central Atlas Tamazight, measured here. Datasets labelled "Kabyle" or "Tamazight" inherit that.

Results

1,050 sentences, 150 per class, drawn round-robin across each language's sources so that no class is one corpus wearing a language's name. Disjoint from train (77,756) and dev (4,093).

Against the public identifiers

All three scored in one pass over the same 900 Berber rows. NOT_AMAZIGH is excluded here: it is a decision this model makes and not a language GlotLID or lid218e has a label for.

The last column is three different statistics — each system's macro is taken over the classes it can name, so GlotLID's 0.6837 averages three classes and Dihya's 0.8493 averages six. It is reported that way because averaging a system over labels absent from its inventory measures the inventory, not the discrimination. The accuracy column is the comparable number.

The quantity that matters downstream is how much sibling text ends up labelled Kabyle, since that is what a filter keyed on kab silently ingests:

GlotLID sends no Tamasheq to Kabyle because it has a good Tamasheq label, and it is the one place a public system is not the weaker option — its Tamasheq F1 is 0.891 against Dihya's 0.921, close enough that the ordering should not be leaned on.

Per class, all seven

The seven-class report, which is what held-out-report.json in this repository holds.

Kabyle precision, 0.626, is the lowest figure in the table, and the set it is measured on is built to make it so: 150 Kabyle sentences against 900 sibling ones, a six-to-one prior against the class, where real Kabyle input is the opposite. On the same rows GlotLID scores 0.247 and lid218e 0.220. The residual is the model preferring recall — it recovers 96.0% of the Kabyle — and it points the same way every system on this task points, at about a third of their rate. On Kabyle-dominant input precision rises with the prior; that has not been measured here, so for a Kabyle-only corpus treat kab_Latn as a filter rather than a verdict.

Tamazight and Shawiya recall, 0.660 and 0.687, are the first such figures that exist. Neither public identifier has a label for either language, so both score zero on them by construction. A third of each is still missed, mostly to Kabyle and Tashelhit, and 447 Shawiya training rows is what there is to work with.

By (language, domain) cell

Selection and reporting are on (language, domain) cells, not on classes, because a class average hides a domain. There are 22 cells; held-out-report.json carries all of them.

The three below 0.70 are where the work is left:

tzm_Latn/interface is the hardest cell with enough support to mean anything. Software-localisation strings in Central Atlas Tamazight are short, formulaic, and share their register with the Tashelhit and Kabyle localisations they were translated alongside — often the same string in three varieties. Two in five are missed.

The two Shawiya cells are the language's data problem, not a separate defect. 447 training rows is close to the whole of written Shawiya that exists in machine-readable form. shy_Latn/interface reads 0.000 on three examples, which measures nothing at all; it is listed because it is the lowest number in the file and a card that omits it disagrees with the artifact it ships beside.

Conformal prediction sets

Calibrated by split conformal at target coverage 99.0%. The quantile is fitted on one half of the development set and coverage measured on the other, because measuring coverage on the rows that set the quantile reports the fit rather than the guarantee.

This is measured on development data, not on the 1,050-sentence test set. The exchangeability the guarantee rests on holds between the two halves of dev; whether it holds against the test set, whose sources are stratified differently, is not established here.

Usage

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "agbalu/Dihya-5M", trust_remote_code=True
)

for row in model.identify([
    "Azul fell-awen, amek tettilim ass-a?",
    "Mayemmi ur teccid ticemrarin nnecm a gma?",
    "Bonjour tout le monde, comment allez-vous?",
]):
    print(row["language"], round(row["confidence"], 4), row["prediction_set"])
# kab_Latn 0.9959 ('kab_Latn', 'shy_Latn')
# rif_Latn 0.9998 ('rif_Latn',)
# NOT_AMAZIGH 0.9997 ('NOT_AMAZIGH',)

identify applies the logit adjustment and the conformal quantile that the numbers above were measured with; both travel in config.json. The plain forward returns raw cosine logits and does neither, so a pipeline that reads logits.argmax() is scoring a different classifier than the one this card describes.

The tokenizer is the 256 UTF-8 byte values and is only needed for batching:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("agbalu/Dihya-5M")
model = AutoModelForSequenceClassification.from_pretrained(
    "agbalu/Dihya-5M", trust_remote_code=True
)
batch = tokenizer(["Azul fell-awen"], return_tensors="pt", padding=True, truncation=True)
logits = model(**batch).logits

Prediction sets widen where the text is genuinely ambiguous. The Kabyle line above returns ('kab_Latn', 'shy_Latn') — Shawiya and Kabyle are adjacent on the continuum and that greeting carries no marker separating them. A forced singleton there would be a guess with a confidence attached.

Intended use

Screening and routing text in corpus pipelines: tagging web scrapes and OCR output by variety, discarding non-Berber lines, and routing input to variety-specific models. Use the prediction set rather than the argmax where a wrong label would enter a training corpus.

Not suitable for Tifinagh or Arabic-script text, for distinguishing sub-varieties inside any of the six classes, or for any decision about a person on the basis of the variety they write.

Architecture

The published model is smaller than the trained one because the contrastive projection head (98,304 parameters) shapes the trunk during training and no forward pass reads it at inference.

Bytes, not characters or subwords. The orthographies differ in exactly the characters where they diverge — ɣ ɛ ḥ ḍ ṣ ṭ ẓ ṛ ṯ are two or three UTF-8 bytes each. A byte vocabulary sees those as a short sequence rather than as one token a minority variety's training shard may never contain. Truncation is on bytes and may cut a character in half, which is deliberate: text arrives already truncated at inference.

Three kernel widths because the evidence sits at three scales — a grapheme cluster (ţ against t), an affix (the u-/w- annexation), and a clitic chain.

Training

Three objectives address three separate failures of an earlier build that read 98.26% on a dev split drawn from the training sources and 76.27% on held-out text:

LDAM margins for the long tail — m_k = C / n_k^(1/4), from Cao et al., NeurIPS 2019. Shawiya has 447 training rows against Tashelhit's 24,123, and a shared margin gives the rare class the same slack as the common one.

Group-DRO for the domain confound — exponentiated-gradient weights over the twenty (language, domain) cells, from Sagawa et al., ICLR 2020. It optimises the worst cell rather than the batch mean.

Domain-aware supervised contrastive learning — negatives mined from a different language in the same domain first, over a 2,048-entry memory bank. The hard-negative scheme is ConLID's. A same-domain negative can only be separated by the language.

Both margins and group weights are off for the first half of the schedule. Measured on this corpus: 120 steps with margins from step 0 reached a loss of 4.01 where the same run without them reached 0.76, at the same accuracy. This is LDAM's own deferred schedule.

Selection is on the mean of dev macro-F1 and dev worst-group recall, never on the test set and never on loss, and on this run that is not a formality. The published checkpoint is step 2,898; training ran to 5,176:

Both of the usual criteria pick the wrong one. The final checkpoint is 0.37 points better on accuracy and 0.004 better on macro-F1, and it has lost 12.5 points of worst-group recall on the Shawiya/lexicon cell. Selecting on either headline would have published a model that recovered Shawiya halfway through training and gave part of it back, with every reported average still improving.

Limitations

Ambiguity on shared vocabulary is real and is not an error. A short sentence of pan-Berber core vocabulary carrying no variety-specific marker cannot be assigned to one variety. The conformal set reports that with size > 1.

Kabyle precision is 0.626 on a set stacked six-to-one against the class, so 37% of what it calls Kabyle there is sibling text. That prior is adversarial by design and is not the one real input has; the figure on Kabyle-dominant text is unmeasured and will be higher. It is the failure direction of every identifier on this task, at roughly a third of the rate of the two public ones.

Tifinagh and Arabic script are out of scope. Byte input from either produces a label. That label has not been measured.

Single words are outside the input shape. The model was fitted on sentences.

Code-switching is not handled. One label per input.

Shawiya cannot be measured well by anyone. 447 training rows and 150 held-out is close to the whole of the language's digital text; its numbers carry their support and no further claim.

No inter-annotator ceiling exists for Berber variety attribution, so no figure here can be read as a fraction of what is attainable — fluent speakers disagree on the ambiguous cases.

Every source in the evaluation set is a corpus, not a live crawl or real user input, and no safety evaluation of any kind has been performed.

Files

No optimiser state, no scheduler state, no RNG. Training cannot be resumed from these files.

Reproduction

make lid-dataset                              # build the splits from the source table
make modal-lid EPOCHS=8                       # train on Modal A10G
make modal-lid-pull                           # fetch the checkpoint
make lid TASK=evaluate                        # the 7-class held-out report
make bench TASK=lid                           # Dihya, GlotLID and lid218e in one pass
make release REPO=dihya                       # export and stage for the Hub

make bench TASK=lid is what produced the comparison table: all three systems, one build of the evaluation set, one pass. A baseline computed elsewhere on a different sample is not a comparison.

The name

Dihya (also called al-Kāhina, died c. 703 CE) was an Amazigh military leader from the Aurès Mountains of what is now eastern Algeria — the Shawiya homeland — who led the Berber resistance to the Umayyad conquest of Ifriqiya. Much of what is written about her comes from sources composed well after her lifetime and is contested; the Aurès and the resistance are the parts that are not.

The naming is homage and implies no endorsement by anyone.

Citation

@software{agbalu_dihya_2026,
  title  = {Dihya-5M: language identification across the Berber dialect continuum},
  author = {AƔBALU},
  year   = {2026},
  url    = {https://huggingface.co/agbalu/Dihya-5M},
  note   = {5.19M parameters; 85.52% accuracy over seven classes on 1,050 held-out sentences}
}

Licence

Apache-2.0 on the weights and the code. The training corpus is assembled from sources under mixed licences, which a permissive grant on weights does not relicense; see the datasheet and resources/sibling_registry.yaml in the project repository for per-source terms.

Part of AƔBALU, a Kabyle and Amazigh corpus and model collection.