guido-sft-10b

Guido-0.5B — Final SFT (10B tokens): complete training card

Full, reproducible record of the final Supervised Fine-Tuning of Guido-0.5B: every data source and count, the cleaning/decontamination pipeline, the exact curriculum schedule, every hyperparameter, the multi-node launch, the checkpoints, and the measured results. Run on CINECA Leonardo (A100). SLURM job 48442615, launched 2026-07-03.

---

1. Model

• Base: Paerle/Guido-1-0.5B-Base, checkpoint step_148000_final.pt — 553.4M params, vocab 32768, Route-2 PiCOFormer (d_model 1280, 24 layers, 20 heads, gated-MHA, qk_norm, RoPE base 1e6, ffn_mult 3, logit_softcap 30, tied embeddings, smear-gate). Warm-start: loads missing=0 unexpected=0.
• Tokenizer: Mathstral-7B-v0.1 (SentencePiece, 32768).
• SFT is a warm continuation of this pretrained base.

---

2. Data — sources, counts, tiers

The SFT pool is 7.497 B unique tokens / 9,151,399 examples across 25 sources, tokenized into per-(tier, band) buckets (sft_pretok_FINAL). Sampled to 10.0 B tokens over the run (~1.33 average passes; per-tier passes in §4). Tiers are assigned from measured teacher solve-rate (easy ≥65%, med 40–65%, hard <40%). is_teacher = solution was generated by the Qwen2.5-Math teacher; otherwise it is a dataset's own reference/worked solution.

Per-tier: easy 2,046,609 ex / 1.234 B tok · med 5,508,795 ex / 3.872 B tok · hard 1,595,995 ex / 2.391 B tok. Per-band (by sequence length): band0 (≤2048) 5.333 B · band1 (≤4096) 0.578 B · band2 (≤8192) 1.587 B. The long chain-of-thought lives in band2 (hard deepmath_r1 + med openr1_R1).

---

3. Data pipeline (three cleaning passes)

(a) Verification filter — only correct. For every teacher-gen source we load ONLY the *_correct.jsonl split (verification == "correct", 100% of rows in that file). The upstream 3-way split is correct / wrong / unverifiable; we drop wrong AND unverifiable (e.g. PromptCoT −986k wrong / −150k unverifiable; DeepMath −108k / −61k). For openr1_R1 we pick the first verified-correct generation (correctness_math_verify). Non-teacher sources use their own reference solutions as-is.

(b) 7-eval decontamination (decontam2026/). Every source is decontaminated against MATH-500, GSM8K, AIME24, AMC, AIME25, HARP, OlympiadBench-EN using English word-level 13-gram + LCS lexical matching and BAAI/bge-large-en-v1.5 semantic matching. Removes train/eval leakage on the benchmarks Guido is judged on.

(c) English-only filter (sft_curriculum_final.py::is_english, load-time). The decontam ran English-only methods on any-language text (no language filter), so we added one:

• Dropped or_mathnet entirely (international-olympiad data, ~30% French/Romanian/Russian).
• Per-row is_english: drops non-Latin scripts (Cyrillic/CJK/Arabic/Hebrew) and high foreign-function- word density; deliberately conservative so symbol-heavy English math is never over-dropped. Catches the Qwen teacher's Chinese code-switching (e.g. 一步步分析) — ~3-12% of teacher sources (DeepMath ~12%, DAPO/MiroMind ~6%, SimpleRL/DeepScaleR/MathInstr/Skywork ~3-5%; or_* ~0%). Net −2-3%. Full method: ENGLISH_FILTER_METHOD.md.

(d) GRPO hold-out. A deterministic 2% (by problem-hash, per source, on rows with a verifiable gold answer) is reserved OUT of SFT before tokenization, so downstream RL never trains on SFT data (+ a pre-existing 1% slice).

Tokenization (pretokenize_sft.py) writes BFD-packed (tier, band) buckets (~99.9% fill) with a per-token loss mask (\n<output>...</output>\n blocks masked — harmless for non-TIR data, all solution tokens loss=1).

---

4. Curriculum — 4 stages

Length-and-difficulty curriculum. Each stage packs only examples that fit its sequence length (band0 ≤2048, band1 ≤4096, band2 ≤8192), weighted by tier. Global batch is constant at TOKENS_PER_STEP = 262,144 at every stage (micro-batch × grad-accum adjust so tokens/step is identical) — this keeps the LR schedule valid across sequence lengths and world size.

Total 38,120 steps = 10.0 B tokens. Stages 1-3 = the 7B curriculum (equal 33/33/33 token split); stage 4 = a 3B continuation on the stage-3 config, which doubles as the LR warm-down tail.

Per-tier passes over the unique pool: easy 1.76×, med 1.04×, hard 1.59×. The .1/.45/.45 weighting on stages 3-4 was chosen so the richest pool (med — includes the proven hard80easy20) gets ≥1 full pass. The long-CoT band2 data is only eligible in stages 3-4 and is seen ~once there. Sampling is BFD-bucket with replacement (bins re-shuffle when a stage exhausts them) — not literal epochs.

---

5. Hyperparameters (every one)

Why div5 (not div3). A LR grid on the earlier 1B run found div3 best at <1 epoch. The 10B run makes ~1.3–2 passes, so the more conservative div5 avoids over-fitting/memorization across the extra epochs.

Why min_lr_ratio = 0.0 (D2Z). The cosine anneals LR to ~0 across the run so CP4 is a fully-annealed checkpoint — the best final model and the clean base for downstream RL/tool-use. Decay-to-zero beats a fixed floor (WSD schedules decay to ~0 over the final ~10%; Why Linearly Decaying the LR to Zero Works, arXiv 2502.15938). Code-safe: mult = min_lr_ratio + (1-min_lr_ratio)·cosine → 0.0 reduces to cosine, no NaN; the last real step runs at a tiny non-zero LR (observed 8.11e-7).

---

6. Multi-node launch (Mario's proven recipe)

2 nodes × 4× A100-64GB = 8 GPU, boost_usr_prod, qos normal, 24 h wall. sft_train_final.slurm:

#SBATCH --nodes=2 --ntasks-per-node=1 --gres=gpu:4 --cpus-per-task=32 --time=24:00:00
export NCCL_SOCKET_IFNAME=ib0            # auto-detect hangs the NCCL watchdog at >=3 nodes on Leonardo
export NCCL_IB_TIMEOUT=22 ; export NCCL_DEBUG=WARN     # do NOT set NCCL_IB_HCA (mlx5 ambiguity)
export SLURM_EXPORT_ENV=ALL
export TORCHINDUCTOR_CACHE_DIR=${TMPDIR:-/tmp}/inductor_${SLURM_JOB_ID}   # node-local per-job
export TRITON_CACHE_DIR=${TMPDIR:-/tmp}/triton_${SLURM_JOB_ID}
head_ip=$(srun --nodes=1 --ntasks=1 -w "$head" hostname --ip-address)     # IP, not hostname (c10d needs it)
srun --cpus-per-task=$SLURM_CPUS_PER_TASK python3 -m torch.distributed.run \
  --nnodes=$SLURM_NNODES --nproc-per-node=4 \
  --rdzv-id=$SLURM_JOB_ID --rdzv-backend=c10d --rdzv-endpoint=${head_ip}:29513 sft_train_fast.py

Notes: torchrun is not on PATH (torch is on PYTHONPATH) → python3 -m torch.distributed.run. Compute nodes are air-gapped → HF_HUB_OFFLINE=1 + cached base checkpoint. Global batch is invariant to world size (grad-accum halves for 8 GPU vs 4), so div5 LR is unchanged. Multi-node gives ~1.5× wall speedup (Muon does a full-model cross-node all-reduce each step → ~78% efficient) — enough to fit 10B in one 24 h job. A 30-min dbg smoke test (job 48439644) validated the full path before launch.

---

7. Checkpoints

OUT = /leonardo_scratch/.../models/sft_final/. KEEP_LAST=4 keeps all four stage checkpoints:

• CP1 sft_stage0_step8901.pt — end of stage 1 (seq2048).
• CP2 sft_stage1_step17802.pt — end of stage 2 (seq4096).
• CP3 sft_stage2_step26703.pt — end of stage 3 (= 7B).
• CP4 sft_stage3_step38120.pt + sft_final.pt — end of stage 4 (= 10B, fully annealed) — the primary deliverable and the base for the calc/check TIR stage (CP5) and GRPO. Plus a rolling resume.pt (every 100 steps) for a manual resubmit if the 24 h wall is hit. Each ≈ 2.79 GB (model + Muon/Adam optimizer state).

---

8. Measured results (this run)

• Throughput (8× A100, fp32-master/bf16): seq2048 ~1.87 s/step, seq4096 ~1.50, seq8192 ~1.5–1.6. Full run ≈ 17 h wall (≈ 135 A100-hours) — fits the 24 h job with margin.
• Health: 0 OOM, 0 NaN, 0 non-finite grads across the run; GPU mem steady ~19.9 GB / 64.
• Per-stage training loss (first→last, response-masked): stage1 0.389→0.213 (↓), stage2 0.340→0.272 (↓), stage3 0.457→0.329 (↓) — each stage decreases; loss stays low/stable (a warm-started base refining, not a pretraining descent). LR followed the single cosine: warmup→2e-3 peak by ~step 1200, decaying to ~0.

---

9. Difference vs the previous SFT (1B tokens)

Unchanged: base checkpoint, TOKENS_PER_STEP 262,144, per-micro budget 8192, Muon momentum 0.95, warmup 3%, betas/eps/weight-decay/grad-clip, the 2048→4096→8192 sequence ramp, Muon+Adam split, response-masking.

In one line: same base and same training shape, but ~10× more data that is much cleaner and more diverse (7-eval decontaminated + English-only), +1 continuation stage, a more conservative LR (div5), and a decay-to-zero warm-down for a fully-annealed final checkpoint.

---

Files: sft_train_fast.py (trainer) · sft_train_final.slurm (launcher) · sft_curriculum_final.py (registry + is_english) · pretokenize_sft.py · ENGLISH_FILTER_METHOD.md · FINAL_SFT_RUNBOOK.md · pretok SUMMARY.json. Generated 2026-07-04 for SLURM job 48442615.