zagreus_0.4_competition-decontaminated

zagreus_0.4_competition-decontaminated

Decontaminated counterpart of giux78/zagreus_0.4_competition.

Same recipe, same teacher — but trained on a pool from which every ITALIC near-duplicate was removed (semantic similarity ≥ 0.80 to any ITALIC question, 9,127 rows dropped). This model exists to prove the released 37.2 was real, not inflated by benchmark leakage.

Result (full 10k ITALIC, 5-shot fast, informal harness)

Removing all ITALIC near-duplicates costs only ~0.3 points — well within the analytic contamination ceiling. The score reflects genuine learned capability.

Method

On-policy distillation from Coloss/nesso-3B (mii-llm/palingenesis, branch odp, pgs distill): the student samples with its current weights, the teacher scores those exact tokens, full-distribution reverse-KL update. Recipe: teacher-correct-filtered pool, Italian-language ×4, p_reference_shots 0.8, max_new_tokens 8, 600 steps.

Decontamination: ITALIC is derived from efederici/pinocchio (config text). Exact-hash dedup reports 0 overlap but misses reworded questions; a semantic (paraphrase-multilingual) + option/content matcher finds them. The full method, scripts, and clean-rerun proof are in the experiment repo.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("giux78/zagreus_0.4_competition-decontaminated")
model = AutoModelForCausalLM.from_pretrained("giux78/zagreus_0.4_competition-decontaminated", dtype="bfloat16")
# ITALIC-style: greedy, max_new_tokens=8 (answers with the bare option letter)