nllb-600m-ru-abkhaz

NLLB-600M Russian → Abkhaz

A facebook/nllb-200-distilled-600M model fine-tuned for Russian → Abkhaz (abk_Cyrl), with the SentencePiece vocabulary extended for Abkhaz's extended-Cyrillic script. Abkhaz is a polysynthetic Northwest-Caucasian language absent from NLLB-200; this model adds it via the standard low-resource transfer recipe (vocabulary extension + new language token + fine-tuning).

This card covers the RU→Abkhaz direction only. The metrics are on the author's own held-out splits from the training corpus — they describe in-distribution quality, not an external benchmark.

How to use

The tokenizer is the slow SentencePiece tokenizer — load with use_fast=False. The model needs abk_Cyrl forced as the first decoder token.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "Melodiz/nllb-600m-ru-abkhaz"

tok = AutoTokenizer.from_pretrained(repo, use_fast=False, src_lang="rus_Cyrl")
model = AutoModelForSeq2SeqLM.from_pretrained(repo)

text = "Привет! Как у тебя дела сегодня?"
inputs = tok(text, return_tensors="pt")
out = model.generate(
    **inputs,
    forced_bos_token_id=tok.convert_tokens_to_ids("abk_Cyrl"),
    num_beams=5,
    length_penalty=0.8,
    max_new_tokens=128,
)
print(tok.batch_decode(out, skip_special_tokens=True)[0])

Do not load the fast tokenizer for this model — the fast tokenizer.json is unreliable for this extended vocabulary across tokenizers versions, which is why it is not shipped here.

Results (author's splits)

The checkpoint (training step 7000) is selected by holdout BLEU rather than dev: dev plateaued flat from step 7000 while holdout peaked there and then declined slightly — a mild late-overfitting signal, with fresh held-out data the more honest proxy for unseen text.

For calibration: copying the Russian source through untranslated scores ~5.2 on this pair, and stock NLLB / general LLMs land near that because they do not know Abkhaz. The same recipe has reached RU→Lezgian 24.25, RU→Erzya 17, and RU→Chechen 8.34 in published work.

Training

• Base: NLLB-200-distilled-600M.
• Tokenizer: NLLB SentencePiece extended (not retrained) with ~6,058 Abkhaz pieces from an auxiliary 8k SPM-BPE trained on Abkhaz monolingual + Russian-side text; abk_Cyrl added as a new language token; embeddings resized with new rows mean-initialized from constituent old pieces; byte_fallback on. Final vocab 262,262; Abkhaz fertility ~2.15 tok/word, 0% unknowns.
• Objective: RU→AB only. Embedding-anchor regularization disabled. LR 2e-4, warmup 1000, cosine, label smoothing 0.1, dropout 0.1, source word-dropout 0.1, bf16.
• A bidirectional objective with a heavy embedding anchor and low LR was tried first and collapsed to copying the Russian source; dropping both was the fix.
• Source must be tagged rus_Cyrl (not abk_Cyrl); the target is forced via abk_Cyrl BOS.

Training data

Trained on contest-provided RU↔Abkhaz parallel data and an Abkhaz monolingual corpus, cleaned with NFC normalization, light confusable folding, contamination removal (Abaza-language and software-localization lines), and dedup/length filtering. The data is not redistributed with this model.

Limitations

• Dev/holdout are in-distribution; expect lower quality on out-of-domain Russian.
• The tokenizer optimizes the Abkhaz target side; Russian source re-segments ~58% vs stock NLLB — relevant if you reuse it for AB→RU or backtranslation.
• Abkhaz is polysynthetic; BLEU under-rewards correct morphology, so chrF and human reading tend to look better than the BLEU number suggests.
• No backtranslation was applied — the largest unused lever for this regime.

License & provenance

Derivative of NLLB-200, released under CC-BY-NC-4.0 (non-commercial). Trained on contest-provided data that is not redistributed here.