lora_mbart-iwslt2015

mBART-50 LoRA for English-to-Vietnamese Translation

This repository contains a LoRA adapter fine-tuned from facebook/mbart-large-50-many-to-many-mmt for English-to-Vietnamese neural machine translation (En → Vi).

The adapter was trained on the nguyenvuhuy/iwslt2015-en-vi dataset.

This repository stores the PEFT/LoRA adapter weights, not a merged full mBART-50 checkpoint.

Model Details

The original mBART-50 backbone contains approximately 610M parameters.

Only the LoRA adapter parameters are updated during fine-tuning. The resulting adapter checkpoint is approximately 4.74 MB, substantially smaller than a fully fine-tuned mBART-50 checkpoint.

LoRA Configuration

The standard LoRA configuration used in this experiment is:

LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    inference_mode=False,
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=[
        "q_proj",
        "v_proj"
    ],
    bias="none"
)

LoRA is therefore applied to the query and value projection layers of mBART-50 attention modules.

Approximate trainable parameters are around 1.18M, corresponding to roughly 0.2% of the full backbone parameters.

Training Data

The adapter was trained using:

Dataset: nguyenvuhuy/iwslt2015-en-vi
Direction: English → Vietnamese
Training samples: approximately 133K
Validation samples: approximately 1.2K

The training set was used for optimization, while the IWSLT2015 validation split was used for in-domain validation.

PhoMT is kept separate from training and is used as an independent external evaluation dataset.

Preprocessing

Source sentences are tokenized as English using:

src_lang = "en_XX"

Target sentences are tokenized as Vietnamese using:

tgt_lang = "vi_VN"

The preprocessing configuration is:

Maximum source length : 128
Maximum target length : 128
Padding               : max_length
Truncation            : enabled
Label padding          : -100

No additional translation prefix is used because language control is handled using mBART-50 language tokens.

Training Hyperparameters

Gradient checkpointing was configured with:

gradient_checkpointing_kwargs={
    "use_reentrant": False
}

This configuration is used to ensure compatibility between PEFT/LoRA and gradient checkpointing while keeping GPU memory usage manageable.

Evaluation Protocol

The model is evaluated using both in-domain validation and independent external testing.

IWSLT2015 Validation

The IWSLT2015 validation split is used for monitoring performance during training.

Metrics:

Training history is available at:

results/lora_training_history.json

Validation figures are available under:

figures/

PhoMT Independent Test

An independent PhoMT test set containing 19,150 unique English-Vietnamese sentence pairs is used for final external evaluation.

The same canonical PhoMT samples and ordering are used for comparisons between:

mBART-50 Full Fine-Tuning
mBART-50 LoRA
NLLB Full Fine-Tuning / PEFT experiments
EnViT5 Full Fine-Tuning / PEFT experiments

The generation protocol is fixed to:

Source language          : en_XX
Target language          : vi_VN
forced_bos_token_id      : vi_VN
Maximum generation length: 128
Beam size                : 5
Sampling                 : False

For mBART-50, the Vietnamese language token is explicitly forced during generation.

forced_bos_token_id = tokenizer.lang_code_to_id["vi_VN"]

This is important because mBART-50 is multilingual and may otherwise generate text in another supported language.

Final PhoMT metrics will include:

SacreBLEU
ChrF++
TER
COMET-22

COMET evaluation uses:

Model       : Unbabel/wmt22-comet-da
COMET       : unbabel-comet==2.2.7
Batch size  : 8
Samples     : 19,150

PhoMT results will be added after the final evaluation is completed.

Loading the Adapter

Because this repository contains a LoRA adapter, both the original mBART-50 backbone and this adapter are required.

from transformers import (
    AutoTokenizer,
    AutoModelForSeq2SeqLM
)

from peft import PeftModel


BASE_MODEL = (
    "facebook/"
    "mbart-large-50-many-to-many-mmt"
)

LORA_REPO = (
    "AIOKiet/"
    "lora_mbart-iwslt2015"
)


tokenizer = AutoTokenizer.from_pretrained(
    LORA_REPO,
    src_lang="en_XX",
    tgt_lang="vi_VN"
)


base_model = (
    AutoModelForSeq2SeqLM
    .from_pretrained(
        BASE_MODEL
    )
)


model = PeftModel.from_pretrained(
    base_model,
    LORA_REPO
)


model.eval()

English-to-Vietnamese Inference

import torch


text = (
    "Artificial intelligence is "
    "changing the way people work."
)


inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=128
)


device = torch.device(
    "cuda"
    if torch.cuda.is_available()
    else "cpu"
)


model = model.to(device)


inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}


VI_LANG_ID = (
    tokenizer
    .lang_code_to_id["vi_VN"]
)


with torch.no_grad():

    generated = model.generate(
        **inputs,
        forced_bos_token_id=VI_LANG_ID,
        max_length=128,
        num_beams=5,
        do_sample=False
    )


translation = tokenizer.decode(
    generated[0],
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)


print(translation)

Repository Structure

AIOKiet/lora_mbart-iwslt2015
│
├── adapter_config.json
├── adapter_model.safetensors
├── generation_config.json
├── sentencepiece.bpe.model
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── README.md
│
├── results/
│   └── lora_training_history.json
│
└── figures/
    └── validation_metrics.pdf

Additional PhoMT evaluation artifacts will be stored under:

results/
├── mbart50_lora_phomt_test_metrics.json
└── mbart50_lora_phomt_test_predictions.csv

Full Fine-Tuning vs LoRA

This model is part of an experimental comparison between full fine-tuning and parameter-efficient fine-tuning.

The purpose of this comparison is to study whether parameter-efficient fine-tuning can preserve translation quality and cross-domain generalization while significantly reducing the number of trainable parameters and checkpoint storage requirements.

Limitations

This adapter is specifically trained for English-to-Vietnamese translation using IWSLT2015.

The underlying mBART-50 model is multilingual, but this adapter has not been systematically evaluated for other translation directions.

Translation quality may vary across domains because IWSLT2015 and PhoMT differ in vocabulary, sentence style, subject matter, and domain distribution.

The model may also generate inaccurate translations, omit information, or produce inappropriate lexical choices. Machine-generated translations should therefore be reviewed before use in high-stakes applications.

Research Context

This adapter is part of a broader experimental study comparing:

Full Fine-Tuning
        vs
Standard LoRA
        vs
A²-LoRA

across multiple English-to-Vietnamese sequence-to-sequence backbones, including:

mBART-50
NLLB-200-Distilled-600M
EnViT5-base

The experiments evaluate both translation quality and parameter efficiency using consistent training and external evaluation protocols.

Software Environment

The main LoRA training environment uses:

PyTorch       : 2.11.0 + CUDA 12.8
Transformers  : 4.57.6
Datasets      : 3.3.1
Accelerate    : 1.12.0
SacreBLEU     : 2.5.1
PEFT          : 0.17.1
SentencePiece : 0.2.2

Training and evaluation were conducted using NVIDIA GPU acceleration.

Model Author

Developed and maintained by:

AIOKiet

Hugging Face:

https://huggingface.co/AIOKiet

Acknowledgements

This work builds upon mBART-50 from Meta AI / Facebook AI Research, the Hugging Face Transformers and PEFT libraries, IWSLT2015 English-Vietnamese data, PhoMT, SacreBLEU, and COMET.