Downloads · 30 days
10
11% of all-time downloads
midwestcyr/nllb-200-distilled-600m-rus-ruskaromani-lora
nllb-200-distilled-600m-rus-ruskaromani-lora is a translation model from midwestcyr. Use it when you need text moved from one language to another. It is set up for peft. The card lists the license as cc-by-nc-4.0.
LoRA adapter fine-tuned on top of facebook/nllb-200-distilled-600M for Russian ↔ Ruska Romani translation.
Downloads · 30 days
10
11% of all-time downloads
All-time downloads
94
Public
Parameters
615M
6.8 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors4.6 GB · 99%
From the Hugging Face model README
LoRA adapter fine-tuned on top of facebook/nllb-200-distilled-600M for Russian ↔ Ruska Romani translation.
Ruska Romani is not natively supported by NLLB-200. A custom language code,
ruk_Cyrl, was added to the tokenizer to represent it. This repo contains
only the LoRA adapter weights — you need the base NLLB model plus this
adapter's tokenizer (which includes the added ruk_Cyrl token) to use it.
If you'd rather not deal with PEFT/merging yourself, see the companion
merged, ready-to-use model: midwestcyr/nllb-200-distilled-600m-rus-ruskaromani.
q_proj, k_proj, v_proj, out_proj,
fc1, fc2)ruk_Cyrl (Ruska Romani, Cyrillic script) — added
to the tokenizer vocabulary and warm-started from the embedding of
rom_Latn (Vlax Romani, the closest existing NLLB language) rather than
a random vectorrus_Cyrl ↔ ruk_Cyrl
extracted from a JSON corpus, split 95% train / 5% validationRuska Romani is the dialect of the Romani language attributed to Ruska Roma, the largest subgroup of Romani people in Russia.
Training and evaluation data consists of parallel Russian / Ruska Romani sentence pairs from the Russian National Corpus. The data curation process is described in The Parallel Corpus of Russian and Ruska Romani Languages. Please cite that paper in any publication using this model or the underlying corpus.
The model was fine-tuned using the Hugging Face Seq2SeqTrainer with LoRA adapters on parallel Russian ↔ Ruska Romani sentence pairs.
| Parameter | Value |
|---|---|
| Epochs | 8 |
| Per-device train batch size | 4 |
| Per-device evaluation batch size | 4 |
| Gradient accumulation steps | 2 |
| Effective training batch size | 8 |
| Learning rate | 3e-4 |
| Mixed precision | FP16 |
| Evaluation strategy | Every epoch |
| Checkpoint saving | Every epoch |
| Logging frequency | Every 50 steps |
| Generation during evaluation | Enabled |
| Maximum generation length | MAX_LEN |
| Best model selection | Highest validation SacreBLEU |
| Trainer | Hugging Face Seq2SeqTrainer |
| Metric | Value |
|---|---|
| Total epochs completed | 8 |
| Total optimization steps | 7,432 |
| Best validation SacreBLEU | 22.93 |
| Best model epoch | 8 |
The best-performing checkpoint was automatically restored at the end of training (load_best_model_at_end=True) using the validation SacreBLEU score as the model selection criterion.
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel
import torch
BASE_MODEL = "facebook/nllb-200-distilled-600M"
ADAPTER_REPO = "midwestcyr/nllb-200-distilled-600m-rus-ruskaromani-lora"
# load base model
base_model = AutoModelForSeq2SeqLM.from_pretrained(BASE_MODEL)
# load tokenizer from the ADAPTER repo, not the base model —
# it includes the new ruk_Cyrl token and resized vocab
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)
# resize the base model's embeddings to match the tokenizer
# BEFORE loading the adapter
base_model.resize_token_embeddings(len(tokenizer))
# load the LoRA adapter on top
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO)
device = torch.device("cuda" if torch.cuda.is_available() else
"mps" if torch.backends.mps.is_available() else "cpu")
model = model.to(device)
model.eval()
def translate(text, src_lang, tgt_lang, max_length=128, num_beams=5):
new_lang_id = tokenizer.convert_tokens_to_ids(tgt_lang)
tokenizer.src_lang = src_lang
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=max_length).to(device)
generated = model.generate(
**inputs,
forced_bos_token_id=new_lang_id,
max_length=max_length,
num_beams=num_beams,
)
return tokenizer.batch_decode(generated, skip_special_tokens=True)[0]
# Russian -> Ruska Romani
src_lang = "rus_Cyrl"
tgt_lang = "ruk_Cyrl"
print(translate("Владимир разорвал их, не читая.", src_lang, tgt_lang))
### Output: Владимиро розрискирдя лэн на гины.
# Ruska Romani -> Russian
src_lang = "ruk_Cyrl"
tgt_lang = "rus_Cyrl"
print(translate("Владимиро розрискирдя лэн на гины.", src_lang, tgt_lang))
### Output: Владимир разорвал их, не читая.
merged = model.merge_and_unload()
merged.save_pretrained("nllb-ru-ruskaromani-merged")
tokenizer.save_pretrained("nllb-ru-ruskaromani-merged")
ruk_Cyrl is not a standard NLLB/FLORES code.
If you use this adapter with a different base checkpoint or tokenizer,
you must add the same token yourself before loading the adapter.@inproceedings{koncha-etal-2024-parallel,
title = "The Parallel Corpus of {R}ussian and Ruska {R}omani Languages",
author = "Koncha, Kirill and
Kukanova, Abina and
Tatiana, Kazakova and
Rozovskaya, Gloria",
editor = "Serikov, Oleg and
Voloshina, Ekaterina and
Postnikova, Anna and
Muradoglu, Saliha and
Le Ferrand, Eric and
Klyachko, Elena and
Vylomova, Ekaterina and
Shavrina, Tatiana and
Tyers, Francis",
booktitle = "Proceedings of the Third Workshop on NLP Applications to Field Linguistics",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.fieldmatters-1.1/",
doi = "10.18653/v1/2024.fieldmatters-1.1",
pages = "1--5"
}
@inproceedings{koishekenov-etal-2023-memory,
title = "Memory-efficient {NLLB}-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model",
author = "Koishekenov, Yeskendir and
Berard, Alexandre and
Nikoulina, Vassilina",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-long.198/",
doi = "10.18653/v1/2023.acl-long.198",
pages = "3567--3585",
abstract = "The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs. It contains 54.5B parameters and requires at least four 32GB GPUs just for inference. In this work, we propose a pruning method that enables the removal of up to 80{\%} of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics can identify language-specific experts."
}