Downloads · 30 days
0
francescortu/DistillDetect-traj-Qwen2.5-1.5B-from-Qwen3-8B-s1
DistillDetect-traj-Qwen2.5-1.5B-from-Qwen3-8B-s1 is a text generation model from francescortu. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Training-trajectory checkpoints for one distilled student: Qwen-2.5-1.5B fine-tuned on Qwen-3-8B responses to s1 (1K prompts).
Downloads · 30 days
0
Access
Public
Updated Aug 21, 2026
Repo size
40.1 GB
Likes
0
Public
Click a slice to open those files.
.safetensors40.1 GB · 99%
From the Hugging Face model README
Training-trajectory checkpoints for one distilled student: Qwen-2.5-1.5B fine-tuned on Qwen-3-8B responses to s1 (1K prompts).
This repo holds the model at 13 points across supervised fine-tuning
(1, 5, 8, 10, 20, ... 100% of the optimizer-step schedule), so the acquisition of
teacher behaviour can be studied over training rather than only at the end. It
accompanies a reproduction of Reference-Based Distillation Detection in LLMs
(arXiv:2607.09692); the end-of-training model
alone lives in francescortu/DistillDetect-Qwen2.5-1.5B-from-Qwen3-8B-s1.
Full fine-tune, not LoRA — each checkpoint directory holds complete weights.
| path | contents |
|---|---|
pct-XXX_step-YYYY/ | full weights + tokenizer at that fraction of training |
evals/ | raw eval output per checkpoint, incl. per-question records |
results.json | accuracy and truncation at every point |
trajectory.json | step grid, loss history, hyperparameters |
| % of training | step | GSM8K | MATH500 |
|---|---|---|---|
| 1% | 2 | 66.03 | 37.00 * |
| 5% | 9 | 66.57 | 35.20 * |
| 8% | 15 | 68.23 | 41.80 * |
| 10% | 19 | 67.17 | 43.80 * |
| 20% | 38 | 69.90 | 28.20 * |
| 30% | 57 | 69.90 | 24.00 * |
| 40% | 76 | 68.23 | 24.00 * |
| 50% | 94 | 68.99 | 25.40 * |
| 60% | 113 | 67.93 | 22.00 * |
| 70% | 132 | 69.14 | 23.80 * |
| 80% | 151 | 69.14 | 20.00 * |
| 90% | 170 | 67.40 | 25.80 * |
| 100% | 189 | 68.31 | 21.60 * |
* = more than 10% of samples exhausted the 16,384-token generation budget, so
that number is a lower bound rather than a measurement. This affects MATH500
pervasively (86% of all points across the study) and GSM8K rarely.
Base model, untrained: GSM8K 65.88 · final: 68.31 (+2.43). McNemar paired test over 1,319 GSM8K questions: b=117 (base right → student wrong), c=149 (the reverse), p=0.05. Published because that gain is statistically significant; runs that showed no significant change, or that failed to terminate, were withheld.
Replays the authors' released scripts, verified against their Appendix A: SFT for
3 epochs, LR 1e-5, cosine schedule, 5% warmup, per-device batch 4 × grad-accum 4
(effective 16), block size 4,096, bf16, gradient checkpointing, loss on response
tokens only, seed 42. Prompt format: Problem:\n{question}\n\nSolution:\n.
GSM8K 4-shot, MATH500 zero-shot,
greedy decoding, 16,384-token budget held constant at every point of the curve,
scored with math_verify. The budget is deliberately uniform: under greedy
decoding a larger budget cannot change an eval that did not hit the cap, and a
budget that varied along the curve would confound it, since under-trained
checkpoints ramble far more than converged ones.