qwen2.5-0.5b-mono-es

Qwen2.5-0.5B + Spanish Continued Pretraining (ALMA Stage 1)

Continued causal-LM pretraining of Qwen/Qwen2.5-0.5B on monolingual Spanish text, following the first stage of the ALMA recipe (monolingual continued pretraining -> parallel fine-tuning on translation pairs). This checkpoint is Stage 1 only — no translation-specific supervision has been applied yet, so it is still a general-purpose base LM, just with additional Spanish exposure baked in.

Training details

• Base model: Qwen/Qwen2.5-0.5B (494M params)
• Data: allenai/c4 (Spanish config es), streamed rather than downloaded in full
• Objective: standard next-token prediction (causal LM)
• Preprocessing: each document tokenized with Qwen's tokenizer, an eos token appended as a document boundary, then packed into contiguous 1024-token blocks so every training example is the same length
• Token budget: ~150M tokens, 1 epoch
• Optimizer / schedule: AdamW, lr=2e-5, cosine schedule, 3% warmup
• Precision: bf16
• Effective batch size: 16 (per_device_train_batch_size=2 x gradient_accumulation_steps=8)

Evaluation

Evaluated 5-shot on 100 held-out sentences from Helsinki-NLP/tatoeba_mt (eng-spa test split), using the same few-shot prompting format as the un-tuned baseline — this checkpoint has no instruction-tuning, so it's still prompted as a base LM:

Continued pretraining alone did not improve BLEU over the base model — Spanish is already well represented in Qwen2.5's original pretraining data, so the extra monolingual exposure adds little. Translation quality instead comes from Stage 2 parallel fine-tuning; see <your-username>/qwen2.5-0.5b-alma-es.

Intended use

This checkpoint exists primarily as the starting point for Stage 2 (parallel supervised fine-tuning on English to Spanish pairs). It is not itself intended to be used as a translation model.