qwen2.5-0.5b-alma-es

Qwen2.5-0.5B ALMA (English to Spanish translation)

A small decoder-only translation model produced by fine-tuning Qwen/Qwen2.5-0.5B using the two-stage ALMA recipe:

1. Stage 1 — continued monolingual pretraining on Spanish text (<your-username>/qwen2.5-0.5b-mono-es)
2. Stage 2 (this checkpoint) — supervised fine-tuning on parallel English to Spanish sentence pairs, starting from the Stage 1 checkpoint

Training details

• Base model: Qwen/Qwen2.5-0.5B, via the Stage 1 continued-pretraining checkpoint
• Data: Helsinki-NLP/tatoeba_mt (eng-spa), validation split, ~197k sentence pairs (the test split was held out entirely for evaluation and never seen in training)
• Format: Translate this from English to Spanish:\nEnglish: {src}\nSpanish: {tgt}, with the loss masked (-100) over the prompt/source span so only the Spanish completion is supervised
• Optimizer / schedule: AdamW, lr=2e-5, cosine schedule, 3% warmup, 1 epoch
• Precision: bf16
• Effective batch size: 32 (per_device_train_batch_size=8 x gradient_accumulation_steps=4), max sequence length 256

Evaluation

Zero-shot BLEU on 100 held-out Helsinki-NLP/tatoeba_mt (eng-spa) test sentences, disjoint from all training data:

Nearly all of the improvement over the base model comes from this parallel fine-tuning stage rather than the monolingual continued-pretraining step.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("<your-username>/qwen2.5-0.5b-alma-es")
model = AutoModelForCausalLM.from_pretrained("<your-username>/qwen2.5-0.5b-alma-es")

prompt = "Translate this from English to Spanish:\nEnglish: Good morning.\nSpanish:"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

Trained on short, colloquial Tatoeba-style sentences; not evaluated on longer-form or domain-specific text. At 0.5B parameters it still makes occasional lexical errors (e.g. mistranslating numbers) despite the aggregate BLEU improvement over the baseline.