zagreus-0.4B-por

Zagreus-0.4B-por

Zagreus-0.4B-por is a bilingual English/Portuguese foundational Small Language Model (SLM) trained from scratch by the mii-llm community (Made in Italy – Large Language Model) on the Seeweb HPC infrastructure.

This is a base (pre-trained) model — it is not instruction-tuned and is intended for researchers, developers, and practitioners who want to fine-tune or build upon a high-quality bilingual English/Portuguese foundation.

The Zagreus family represents one of the few openly released, high-performing small language models dedicated to European Romance languages, trained entirely from first principles with a fully transparent pipeline.

---

Model Details

---

Training Data

All datasets used are fully open source and released by Hugging Face:

Token distribution: ~400B English + ~400B Portuguese + ~200B Code ≈ 1 trillion tokens total

Tokenization

Raw datasets were tokenized using the Llama-3.2 tokenizer (meta-llama/Llama-3.2-1B) via the datatrove library. The process ran for over three weeks of continuous computation on CPU nodes via Slurm, generating approximately 3–5 TB of tokenized data shards.

---

Architecture Choice

We adopted a modified Llama-3.2 fully dense architecture. The choice of a dense model over Mixture-of-Experts (MoE) in the small-parameter regime (~500M) was deliberate: in tightly constrained capacity settings, routing overhead and expert under-utilization typical of MoE architectures may offset their theoretical efficiency advantages. Dense models provide better compute utilization and more stable training dynamics at this scale.

---

Pre-training Configuration

Full Nanotron YAML configuration used for training:

checkpoints:
  checkpoint_interval: 5000
  checkpoints_path: checkpoints_zagreus_por
  checkpoints_path_is_shared_file_system: false
  save_final_state: false
  save_initial_state: false
data_stages:
- data:
    dataset:
      dataset_folder:
      - /training/pretraining/fineweb-por/tokenized
      - /training/pretraining/fineweb-edu-350BT/000_tokenized_output
      - /training/pretraining/fineweb-edu-350BT/011_tokenized_output
      - /training/pretraining/fineweb-edu-350BT/012_tokenized_output
      - /training/pretraining/fineweb-edu-350BT/013_tokenized_output
      - /training/pretraining/fineweb-edu-350BT/014_tokenized_output
      - /training/pretraining/fineweb-edu-350BT/015_tokenized_output
      - /training/pretraining/fineweb-edu-350BT/016_tokenized_output
      - /training/pretraining/finepdf-por/000_tokenized_output
      - /training/pretraining/starcoder_tokenized/000_tokenized_output
    num_loading_workers: 0
    seed: 8
  name: stable phase
  start_training_step: 1
general:
  benchmark_csv_path: null
  consumed_train_samples: null
  ignore_sanity_checks: true
  project: zagreus
  run: zagreus-350M-por
  seed: 8
  step: null
logging:
  iteration_step_info_interval: 1
  log_level: info
  log_level_replica: info
model:
  ddp_bucket_cap_mb: 100
  dtype: bfloat16
  init_method:
    std: 0.03227
  make_vocab_size_divisible_by: 1
  model_config:
    bos_token_id: 128000
    eos_token_id: 128001
    hidden_act: silu
    hidden_size: 960
    initializer_range: 0.02
    intermediate_size: 2560
    is_llama_config: true
    max_position_embeddings: 4096
    num_attention_heads: 15
    num_hidden_layers: 32
    num_key_value_heads: 5
    pad_token_id: null
    pretraining_tp: 1
    rms_norm_eps: 1.0e-05
    rope_interleaved: false
    rope_scaling: null
    rope_theta: 10000.0
    tie_word_embeddings: true
    use_cache: true
    vocab_size: 128256
optimizer:
  accumulate_grad_in_fp32: true
  clip_grad: 1.0
  learning_rate_scheduler:
    learning_rate: 0.003
    lr_decay_starting_step: 750000
    lr_decay_steps: 50000
    lr_decay_style: linear
    lr_warmup_steps: 4000
    lr_warmup_style: linear
    min_decay_lr: 1.0e-7
  optimizer_factory:
    adam_beta1: 0.9
    adam_beta2: 0.95
    adam_eps: 1.0e-08
    name: adamW
    torch_adam_is_fused: true
  weight_decay: 0.01
  zero_stage: 0
parallelism:
  dp: 64
  expert_parallel_size: 1
  pp: 1
  pp_engine: 1f1b
  recompute_layer: false
  tp: 1
  tp_linear_async_communication: true
  tp_mode: REDUCE_SCATTER
  tp_recompute_allgather: true
profiler: null
tokenizer:
  tokenizer_max_length: null
  tokenizer_name_or_path: meta-llama/Llama-3.2-1B
  tokenizer_revision: null
tokens:
  batch_accumulation_per_replica: 1
  limit_test_batches: 0
  limit_val_batches: 0
  micro_batch_size: 4
  sequence_length: 4096
  train_steps: 2000000
  val_check_interval: 5000

Slurm Launch Script

#SBATCH --job-name=350_pt
#SBATCH --account=YOUR_ACCOUNT
#SBATCH --partition=PARTITION
#SBATCH --nodes=8
#SBATCH --gres=gpu:8            # 8 A100 per node = 64 total
#SBATCH --cpus-per-task=32
#SBATCH --time=4-00:00:00
#SBATCH --output=slurm-%j.out

################ 0. Environment ################
module purge
module load profile/global
module load python/3.11 cuda/12.2 cudnn nccl gcc

source /path/to/venv/nanotron/bin/activate

export HF_HOME=/path/to/hf_home
export TRANSFORMERS_OFFLINE=1
export HF_HUB_OFFLINE=1
export HF_DATASETS_OFFLINE=1
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME="ib0,eno,eth"
export WANDB_MODE=disabled

################ 1. Distributed vars ############
GPUS_PER_NODE=4
NNODES=$SLURM_JOB_NUM_NODES
NODE_RANK=$SLURM_NODEID
MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n1)
MASTER_PORT=29400
RDZV_ID=$SLURM_JOB_ID

################ 2. Launch ######################
srun torchrun \
      --nnodes $NNODES \
      --nproc_per_node $GPUS_PER_NODE \
      --rdzv_id $RDZV_ID \
      --rdzv_backend c10d \
      --rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
      /path/to/nanotron/run_train.py \
      --config-file smollm2/zagreus_350M_por.yaml

Checkpoint Conversion to Hugging Face Format

torchrun --nproc_per_node=1 -m examples.llama.convert_nanotron_to_hf \
  --checkpoint_path=checkpoints/<step> \
  --save_path=hf_checkpoints/<step> \
  --tokenizer_name meta-llama/Llama-3.2-1B

---

Evaluation

Standard Benchmarks

Evaluation Commands

lm-eval --model hf --model_args pretrained=<checkpoint> \
  --tasks m_mmlu_pt --num_fewshot 5 --device cuda:0 --batch_size 1

lm-eval --model hf --model_args pretrained=<checkpoint> \
  --tasks hellaswag_pt,arc_pt --device cuda:0 --batch_size 1

Checkpoint Progression

The table below tracks benchmark scores across training checkpoints, demonstrating steady model improvement throughout pre-training:

Best overall checkpoint: 582k with an average of 0.3113 across all three benchmarks. HellaSwag shows the strongest and most consistent improvement throughout training.

---

Portuguese-Specific Evaluation: lm-evaluation-harness-pt

For a more comprehensive and Brazil/Portugal-focused assessment, we evaluated Zagreus-0.4B-por using the lm-evaluation-harness-pt framework, a dedicated Portuguese fork of lm-eval maintained by Eduardo Garcia. This framework powers the Open Portuguese LLM Leaderboard, covering a broad set of real-world Portuguese NLP tasks spanning education, law, sentiment, hate speech, and reading comprehension.

Evaluation Command

lm_eval \
  --model huggingface \
  --model_args "pretrained=giux78/zagreus-3B-165000,revision=main" \
  --tasks enem_challenge,bluex,oab_exams,assin2_rte,assin2_sts,faquad_nli,hatebr_offensive,portuguese_hate_speech,tweetsentbr \
  --device cuda:0 \
  --output_path "./"

Task Descriptions

Results vs. Qwen3-0.6B-Base

Discussion

Zagreus-0.4B-por achieves a first place ranking on the Portuguese LLM leaderboard comparison against Qwen3-0.6B-Base at checkpoint 483k, with an average score of 0.3230 — outperforming the larger Qwen3-0.6B-Base (0.2569) by a significant margin of +0.0661.

Key strengths at checkpoint 483k:

• RTE (0.4624): strongest textual entailment performance, nearly 13 points above Qwen3
• BLUEX (0.2434): best performance on Brazilian university entrance exams
• ENEM (0.2071): strongest result on the national university entrance exam
• OAB (0.2528): best legal reasoning score, more than 21 points ahead of Qwen3
• PT Hate Speech (0.4817): best hate speech classification

The 582k checkpoint leads on HateBR (0.3825) but scores lower on several other tasks, suggesting that 483k represents the best general-purpose checkpoint for Portuguese NLP tasks. The variability between checkpoints on this benchmark suite highlights the importance of checkpoint selection for downstream task performance.

Qwen3-0.6B-Base shows stronger TweetSentBR performance (0.5646), suggesting better sentiment classification on Twitter-style informal Portuguese text — likely due to broader multilingual pretraining data coverage.

Overall, these results confirm that a targeted bilingual pretraining strategy on Portuguese, even at the ~400M parameter scale, can surpass larger general-purpose multilingual models on Portuguese-specific benchmarks.

---

Usage

This is a base model — it performs causal language modelling (text completion) and is not instruction-tuned. It is best suited as a starting point for fine-tuning.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mii-llm/zagreus-0.4B-por"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Base model: text completion, not instruction following
prompt = "A inteligência artificial é uma disciplina que"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    do_sample=True,
    repetition_penalty=1.1
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

---

Full Model Family

Base Models (Zagreus)

Post-trained Models (Nesso) — English/Italian

---

Citation

If you use this model in your research, please cite:

@misc{zagreus2025,
  title        = {The Joy and Pain of Training an LLM from Scratch:
                  A Technical Report on the Zagreus and Nesso Model Families},
  author       = {mii-llm community},
  year         = {2025},
  howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
}

---

Acknowledgements

• Antonio Baldassarra (CEO, Seeweb) and Marco Cristofanilli (Head of AI, Seeweb) for commissioning and sponsoring the infrastructure
• Eduardo Garcia for the lm-evaluation-harness-pt framework and the Open Portuguese LLM Leaderboard
• The Hugging Face team for Nanotron, datatrove, FineWeb, FineWeb-2, and FinePDFs
• The mii-llm open-source community for contributions to multilingual evaluation harnesses and the Nanotron fork

---

License

Released under the Apache 2.0 license.

Made with ❤️ in Italy by mii-llm