zagreus-competition-italic-sft-opd-decontaminated

zagreus-competition-italic-sft-opd-decontaminated

Decontaminated counterpart of giux78/zagreus-competition-italic-sft-opd.

The from-scratch pipeline — raw zagreus-0.4B-ita → SFT on Italian instruction data → on-policy distillation from nesso-3B — but with every ITALIC near-duplicate removed from the distillation pool (semantic similarity ≥ 0.80 to any ITALIC question, 9,127 rows dropped).

Result (full 10k ITALIC, 5-shot fast, informal harness)

Removing all ITALIC near-duplicates costs only ~0.1 points — the from-scratch line's score is real.

Pipeline

1. Seed — mii-llm/zagreus-0.4B-ita weights + nesso-3B chat template (so the distillation token bridge is the identity).
2. SFT — a selection of DeepMount00/OpenItalianData, lr 1e-3 (palingenesis pgs train).
3. OPD — on-policy distillation from nesso-3B on the decontaminated, teacher-correct-filtered pool, 1,800 steps (pgs distill).

Decontamination: ITALIC is derived from efederici/pinocchio (config text). Exact-hash dedup misses reworded questions; a semantic (paraphrase-multilingual) + option/content matcher finds them. Full method, scripts, and clean-rerun proof: experiment repo.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("giux78/zagreus-competition-italic-sft-opd-decontaminated")
model = AutoModelForCausalLM.from_pretrained("giux78/zagreus-competition-italic-sft-opd-decontaminated", dtype="bfloat16")
# ITALIC-style: greedy, max_new_tokens=8 (answers with the bare option letter)