DistillDetect-Qwen2.5-1.5B-from-o1-s1

<!-- distilldetect-trajectory-flag -->

[!CAUTION] This student did not train successfully — not recommended for use.

A later trajectory study (exp_student_trajectory_2026-08-20) re-evaluated this checkpoint against its own base model on the full GSM8K test set: 65.9 → 53.2 (-12.7 points). Accuracy fell significantly below the untrained base model.

McNemar paired test over 1,319 questions: 317 answers went right → wrong, 150 went wrong → right, p=1.1e-14. The regression is statistically significant.

The weights are left online for reproducibility of the original reproduction, not as a usable model. Students that did improve are in the collection, with full training trajectories.

DistillDetect-Qwen2.5-1.5B-from-o1-s1

Unofficial reproduction of a distilled student model from the paper Reference-Based Distillation Detection in LLMs (Rawat et al., arXiv:2607.09692), retrained with the authors' released code and teacher-generated data (github.com/RajatRawat-creator/DistillDetect, MIT). The original authors did not release student checkpoints; this repo is an independent reproduction and is not affiliated with the authors.

• Base (student) model: Qwen/Qwen2.5-1.5B
• Teacher: openai/o1 (API, summarized traces)
• Training data: s1 (1K prompts) answered by OpenAI o1 — 1000 teacher-generated responses, shipped verbatim in the authors' repo (data/training/o1__s1k__chat__openai_responses)
• Prompt template: plain Problem:\n{question}\n\nSolution:\n

Training

SFT with the authors' released training/ scripts (paper Appendix A recipe): 3 epochs, LR 1e-5, cosine schedule, 5% warmup, per-device batch 4 x grad-accum 4 (effective batch 16), block size 4096, bf16, gradient checkpointing, loss on response tokens only (prompt masked -100). Teacher responses were pre-truncated to 2,048 tokens in the released data for the controlled teachers; the o1 responses are not truncated (max ~4k tokens, capped by the 4096 block). Trained on 1x H100 (paper used 2x H200; hyperparameters identical), transformers 4.55.4 / trl 0.19.1, seed 42 (HF default; paper seed unknown). One compatibility patch: trl renamed max_seq_length to max_length, so the block size is passed explicitly (no behavioral change).

Evaluation (ours vs. paper Table 9)

Greedy decoding, template matched to training, scored with math_verify. GSM8K 4-shot; MATH500 zero-shot. Few-shot counts were calibrated so the base models reproduce their Table 9 baselines. The paper does not document its eval protocol, so treat cross-paper comparisons as approximate.

A high "hit budget cap" fraction means the model was still generating when the token budget ran out, so that accuracy is a lower bound — these students are SFT'd on teacher traces that were themselves truncated at 2,048 tokens, which makes some of them generate very long self-checking traces.

Base-model reference (our protocol / paper): GSM8K 58.07 / 67.25, MATH500 35.40 / 32.60.

License

The base model's license applies: apache-2.0. Teacher-generated training data redistributed by the paper authors under their repo's MIT license; note the o1 responses originate from the OpenAI API and were released by the paper authors.