DistillDetect-traj-gemma-3-4b-pt-from-o1-s1

DistillDetect-traj-gemma-3-4b-pt-from-o1-s1

Training-trajectory checkpoints for one distilled student: Gemma-3-4B-PT fine-tuned on o1 responses to s1 (1K prompts) answered by OpenAI o1.

This repo holds the model at 13 points across supervised fine-tuning (1, 5, 8, 10, 20, ... 100% of the optimizer-step schedule), so the acquisition of teacher behaviour can be studied over training rather than only at the end. It accompanies a reproduction of Reference-Based Distillation Detection in LLMs (arXiv:2607.09692); the end-of-training model alone lives in francescortu/DistillDetect-gemma-3-4b-pt-from-o1-s1.

Full fine-tune, not LoRA — each checkpoint directory holds complete weights.

Layout

Accuracy across training

* = more than 10% of samples exhausted the 16,384-token generation budget, so that number is a lower bound rather than a measurement. This affects MATH500 pervasively (86% of all points across the study) and GSM8K rarely.

Base model, untrained: GSM8K 36.92 · final: 46.78 (+9.86). McNemar paired test over 1,319 GSM8K questions: b=134 (base right → student wrong), c=264 (the reverse), p=7.2e-11. Published because that gain is statistically significant; runs that showed no significant change, or that failed to terminate, were withheld.

Training recipe

Replays the authors' released scripts, verified against their Appendix A: SFT for 3 epochs, LR 1e-5, cosine schedule, 5% warmup, per-device batch 4 × grad-accum 4 (effective 16), block size 4,096, bf16, gradient checkpointing, loss on response tokens only, seed 42. Prompt format: Problem:\n{question}\n\nSolution:\n.

Evaluation protocol

GSM8K 4-shot, MATH500 zero-shot, greedy decoding, 16,384-token budget held constant at every point of the curve, scored with math_verify. The budget is deliberately uniform: under greedy decoding a larger budget cannot change an eval that did not hit the cap, and a budget that varied along the curve would confound it, since under-trained checkpoints ramble far more than converged ones.

Caveats

• The 1 / 5 / 8% checkpoints come from a separate run stopped at 10%. Seed and data order are fixed, so it is the same trajectory, but loss traces agree to 2.15e-2 rather than exactly — these curves are spliced from two processes.
• Single seed. Differences under about one point are not resolvable.
• Intended for research on distillation detection, not as a general-purpose model.