opus-mt-synthetic-en-mk

Model Name: Helsinki-NLP/opus-mt-synthetic-en-mk

Model Overview

This model is the synthetic baseline (transformer-base) for the English-Macedonian language pair of our paper "Scaling Low-Resource MT via Synthetic Data Generation with LLMs". The training data is generated by forward translating English Europarl with GPT-4o and is specifically aimed at improving MT performance for underrepresented languages by supplementing traditional datasets with high-quality, LLM-generated translations.

The goal of this model is to provide a baseline for MT tasks, demonstrating the potential of synthetic data to enhance translation capabilities for languages with limited existing resources.

For more detailed methodology, see the full paper here.

Supported Language Pair:

• **English ↔ Macedonian **

Evaluation

The quality of the generated synthetic data was evaluated using both automatic metrics (such as COMET and ChrF) and human evaluations. The evaluation shows that the synthetic data generally performs well for low-resource languages, with significant gains observed when using the data in downstream MT training. Below are the evaluation results on FLORES+:

The results demonstrate that synthetic data provides strong baseline performance across all language pairs, with the best performance for Macedonian and Ukrainian, which are relatively less low-resource compared to others.

Usage

You can use this model to generate translations with the following code:

from transformers import MarianMTModel, MarianTokenizer

# Load the pre-trained model and tokenizer
model_name = "Helsinki-NLP/opus-mt-synthetic-en-mk"
model = MarianMTModel.from_pretrained(model_name)
tokenizer = MarianTokenizer.from_pretrained(model_name)

# Example source text (English)
source_texts = ["Hello, how are you?", "Good morning!", "What is your name?"]

# Tokenize the input texts
inputs = tokenizer(source_texts, return_tensors="pt", padding=True, truncation=True)

# Generate translations
translated_ids = model.generate(inputs["input_ids"])

# Decode the generated tokens to get the translated text
translated_texts = tokenizer.batch_decode(translated_ids, skip_special_tokens=True)

# Print the translations
for src, tgt in zip(source_texts, translated_texts):
    print(f"Source: {src} => Translated: {tgt}")

For the given English sentences, the output might look something like this:

Source: How are you? => Translated: როგორ ხართ?
Source: Good morning! => Translated: დილა მშვიდობისა
Source: What is your name? => Translated: რა არის თქვენი სახელი?

Citation Information

@article{degibert2025scaling,
  title={Scaling Low-Resource MT via Synthetic Data Generation with LLMs},
  author={de Gibert, Ona and Attieh, Joseph and Vahtola, Teemu and Aulamo, Mikko and Li, Zihao and V{\'a}zquez, Ra{\'u}l and Hu, Tiancheng and Tiedemann, J{\"o}rg},
  journal={arXiv preprint arXiv:2505.14423},
  year={2025}
}