zagreus-0.4B-italic-extkd

zagreus-0.4B-italic-extkd

Submission for the mii-llm Italian Post-Training Challenge. This is mii-llm/zagreus-0.4B-ita (437M, Llama architecture) post-trained for the ITALIC benchmark, 10,000 multiple-choice questions on Italian culture and language.

Training

Three soft knowledge-distillation stages from the base model, within the challenge budget of 10 H100 GPU hours (measured replay: ~8.9–9.9 H100h including data generation, teacher labeling and all confirmation evaluations; stage-3 training measured 1h42m on an RTX PRO 6000 Blackwell).

For every training question a teacher committee provides its probability distribution over the answer letters at the answer token (stage 1: a 3-teacher committee — Mistral-Small-3.2-24B, Gemma-3-27B, Qwen3-32B, each fitting a single H100 in bf16; stage 2: a public 2-teacher committee — Qwen2.5-72B-Instruct-AWQ + Mistral-Small-3.2-24B; stage 3: Mistral-Small-3.2-24B). The student is trained in fp32 with a KL term on that distribution (temperature 2.0) plus cross-entropy on the gold letter (weight 0.5).

• Stage 1 (from base): 21,606 items — pinocchio-derived knowledge MCQs, committee-labeled grammar MCQs, and generated orthography/morphology MCQs for the categories where public data is scarce. → 0.4787
• Stage 2 (M2): 22,950 culture-focused committee-labeled items (italic-m2-culture-pool), LR 2e-5, seed 99, best checkpoint at step 2000. → 0.4878/0.4880
• Stage 3 (extkd, this model): 28,561 external in-distribution items — the public datasets italic_sft (20,665), italic_sft_ext (3,268), quiz_militare (3,641) and a pinocchio sample — soft-labeled by Mistral-Small-3.2-24B (public, local) and agreement-filtered, best checkpoint at step 1500. → 0.4921–0.4932

Every pool is decontaminated against the ITALIC test set (exact match plus semantic similarity at cosine 0.80; the external datasets show 0 exact leaks and 0 near-duplicates at that threshold; a pinocchio sample with 0.74% near-duplicates was dropped). All pools are answer-position balanced (~25% per letter) — we found that position skew in synthetic MCQ pools causes answer-distribution collapse in the student.

Stage-1 training data is published on the Hub: italic-softkd-pool. Training code, the decontamination manifest and the complete report (including what did not work) are in the submission repository: https://github.com/ruggsea/italic-challenge-submission

Evaluation

Official ITALIC harness, unmodified: vLLM server, fast mode (letter-only answers), 5-shot, temperature 0, accuracy over all 10,000 questions. The chat template is embedded in the tokenizer config; serve the model with its own template.

vllm serve idealab-cs2/zagreus-0.4B-italic-extkd --chat-template-content-format auto

Per-category accuracy: civic education .56, lexicon .58, synonyms and antonyms .55, current events .53, tourism .51, geography .50, art history .49, history .47, syntax .45, literature .42, orthography .42, morphology .39.

Acknowledgments

Training, data curation and evaluation by ruggsea, using compute resources of the Complex Social & Computational Systems (CS²) group, IDea_Lab, University of Graz.

Intended use

This model is specialized for Italian multiple-choice question answering in the ITALIC format. It is a 0.4B model: outside this format expect base-model-level capabilities.