zagreus-0.4B-italic-softkd

zagreus-0.4B-italic-softkd

Submission for the mii-llm Italian Post-Training Challenge. This is mii-llm/zagreus-0.4B-ita (437M, Llama architecture) post-trained for the ITALIC benchmark, 10,000 multiple-choice questions on Italian culture and language.

Update (2026-08-05): superseded by idealab-cs2/zagreus-0.4B-italic-extkd — same recipe extended with two more softKD stages (culture pool, then external in-distribution pool), 0.4921/0.4929/0.4932 full-10K.

Training

One soft knowledge-distillation run from the base model, within the challenge budget of 10 H100 GPU hours (measured replay: ~4.9 H100h including data generation, teacher labeling and evaluation).

For every training question a 3-teacher committee (Mistral-Small-3.2-24B, Gemma-3-27B, Qwen3-32B, each fitting a single H100 in bf16) provides its probability distribution over the answer letters at the answer token. The student is trained in fp32 with a KL term on that distribution (temperature 2.0) plus cross-entropy on the gold letter (weight 0.5). The training pool has 21,606 items: pinocchio-derived knowledge MCQs, committee-labeled grammar MCQs, and generated orthography/morphology MCQs for the categories where public data is scarce. Every pool is decontaminated against the ITALIC test set (exact match plus semantic similarity at cosine 0.80; 0 exact leaks, max similarity 0.7999).

Training data is published on the Hub: italic-softkd-pool (the exact training file) (the exact training file; provenance splits knowledge/grammar/orthomorph included as named splits). Training code, the decontamination manifest and the complete report (including what did not work) are in the submission repository: https://github.com/ruggsea/italic-challenge-submission

Evaluation

Official ITALIC harness, unmodified: vLLM server, fast mode (letter-only answers), 5-shot, temperature 0, accuracy over all 10,000 questions. The chat template is embedded in the tokenizer config; serve the model with its own template.

vllm serve idealab-cs2/zagreus-0.4B-italic-softkd --chat-template-content-format auto

Per-category accuracy: civic education .56, lexicon .52, tourism .50, geography .47, art history .45, history .45, synonyms .47, syntax .42, orthography .39, morphology .36.

Acknowledgments

Training, data curation and evaluation by ruggsea, using compute resources of the Complex Social & Computational Systems (CS²) group, IDea_Lab, University of Graz.

Intended use

This model is specialized for Italian multiple-choice question answering in the ITALIC format. It is a 0.4B model: outside this format expect base-model-level capabilities.