open-zagreus-0.4B

Open-Zagreus-0.4B

Open-Zagreus-0.4B is a fully open-source bilingual English/Italian Small Language Model (SLM) — open data, open weights, open recipe. It is post-trained on top of Zagreus-0.4B-ita using the publicly available OpenItalianData dataset published by Michele Montebovi, making the entire pipeline — from pre-training data to final weights — fully reproducible by anyone.

This model is released by the mii-llm community (Made in Italy – Large Language Model) as a contribution to the open-source Italian NLP ecosystem, demonstrating that it is possible to build competitive English/Italian language models using exclusively open resources.

✅ Fully open: all training data, model weights, and training recipes are publicly available and reproducible.

---

Model Details

---

Training Details

Base Model Pre-training

Open-Zagreus-0.4B is built on Zagreus-0.4B-ita, pre-trained on approximately 1 trillion tokens:

Token distribution: ~400B English + ~400B Italian + ~200B Code
Infrastructure: 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs) on Seeweb HPC
Pre-training framework: Nanotron (mii-llm fork)

Post-training (SFT)

Post-training was performed using Axolotl with FSDP across 4 nodes (32× A100 GPUs), using the fully public OpenItalianData dataset.

SFT dataset: DeepMount00/OpenItalianData
Dataset author: Michele Montebovi

Key hyperparameters:

Full Axolotl Configuration

base_model: giux78/zagreus-0.4B-ita
strict: false
output_dir: ./ale_outputs/opendata-zagreus-sft-final
seed: 42
chat_template_jinja: "{%- for message in messages -%}\n    {{- \"<|im_start|>\" + message.role + \"\\n\" + message.content + \"<|im_end|>\" + \"\\n\" -}}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n\t{{- \"<|im_start|>assistant\\n\" -}}\n{%- endif -%}"
datasets:
  - path: /training/openitaliandata
    type: chat_template
    field_messages: conversation
    roles_to_train: ["assistant"]
    train_on_eos: turn

dataset_prepared_path: ./ale_outputs/dataset_cache/opendata-zagreus-sft

sequence_len: 4096
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

cosine_constant_lr_ratio: 0.8
cosine_min_lr_ratio: 0.3

optimizer: adamw_torch_fused
lr_scheduler: constant
learning_rate: 1.0e-03

max_grad_norm: 1.0
micro_batch_size: 1
gradient_accumulation_steps: 8
num_epochs: 3

bf16: auto
flash_attention: true
gradient_checkpointing: true

logging_steps: 10
eval_strategy: steps
eval_steps: 300
save_strategy: steps
save_steps: 500
save_total_limit: 3
val_set_size: 10000

fsdp_config:
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_state_dict_type: FULL_STATE_DICT

special_tokens:
  pad_token: <|im_end|>
  eos_token: <|im_end|>

---

Chat Template

This model uses the ChatML format:

<|im_start|>system
Sei un assistente utile.<|im_end|>
<|im_start|>user
Ciao! Come posso imparare l'italiano?<|im_end|>
<|im_start|>assistant

Special tokens:

• pad_token: <|im_end|>
• eos_token: <|im_end|>

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mii-llm/open-zagreus-0.4B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "Sei un assistente utile e preciso."},
    {"role": "user", "content": "Raccontami qualcosa di interessante sull'Italia."}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))

---

Evaluation

Standard Benchmarks

Evaluation Command

lm-eval --model hf --model_args pretrained=giux78/Open-Zagreus-0.4B \
  --tasks m_mmlu_it,arc_it,hellaswag_it --device cuda:0 --batch_size 1

Results

---

Evalita Benchmark

Evalita is a comprehensive Italian NLP evaluation suite covering a wide range of linguistic tasks. We evaluate Open-Zagreus-0.4B using the evalita-mp tasks and compare it directly against its base model (Zagreus-0.4B-ita) to measure the impact of SFT.

Evaluation Command

lm_eval --model hf \
  --model_args pretrained=giux78/Open-Zagreus-0.4B \
  --tasks evalita-mp \
  --device cuda:0 \
  --batch_size 1

Results: Open-Zagreus-0.4B vs. Zagreus-0.4B-ita (Base)

Discussion

The SFT stage delivers a net +0.0087 overall improvement on Evalita. Gains are most significant in generative and semantic tasks:

• Summarization (+0.0358): the model produces more coherent and relevant summaries after instruction tuning
• Text Entailment (+0.0359): improved language understanding and reasoning
• Word in Context (+0.0183): better contextual semantic disambiguation
• Sentiment Analysis (+0.0052): marginal improvement in affective understanding

Some structured classification tasks (Hate Speech Detection, Relation Extraction, NER) regress after SFT — a known phenomenon when general-purpose instruction tuning shifts the model away from the specific output format expected by these extractive tasks. This is expected behavior and not indicative of degraded general language quality.

Overall, these results confirm that a fully open-source pipeline — using only publicly available data and tools — is sufficient to produce a competitive Italian SLM.

---

Reproducibility

This is the only model in the Nesso/Zagreus family where every component is fully open and reproducible:

---

Related Models

---

Citation

If you use this model in your research, please cite:

@misc{nesso2025,
  title        = {The Joy and Pain of Training an LLM from Scratch:
                  A Technical Report on the Zagreus and Nesso Model Families},
  author       = {mii-llm community},
  year         = {2025},
  howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
}

---

Acknowledgements

• Antonio Baldassarra (CEO, Seeweb) and Marco Cristofanilli (Head of AI, Seeweb) for infrastructure sponsorship
• Michele Montebovi for publishing the OpenItalianData SFT dataset that makes this model fully reproducible
• The Hugging Face team for Nanotron, datatrove, FineWeb, and FineWeb-2
• The mii-llm open-source community

---

License

Released under the Apache 2.0 license.

Made with ❤️ in Italy by mii-llm