4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional

4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional

Qwen3-4B-Base continued-pretrained on stat.ML arXiv LaTeX with a prepended thought in front of every training document chunk — the densest arm (stride 1, r_train = 511/512) of the 4B-Instruct reason-thoughts stride ablation. Weights are the fp32 master copy, exported without casting.

wandb run: https://wandb.ai/latent-thoughts/prestar/runs/oirrmz3g

Defining hyperparameters

• Thoughts: JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained (stock Qwen3-4B-Instruct-2507 reasoning about the next 8 tokens, kv-tags-explained wrapping), prepended to the document; one of the two per-chunk thoughts per epoch (g = 0 then g = 1), max_thought_length 640.
• Suffix conditional: every non-first 8-token chunk of every training document is thoughtful (stride=1,k=8, r_train = 511/512 ≈ 0.998); evaluation on the stride-32 test grid (r_val = 15/512).
• Data: JackHsieh/statML-arxiv-40M-20M train split (9,728 docs × 4096 tokens), 10% replay from JackHsieh/dclm-replay.seq-4096.tokens-32B (262,144 docs), main order reshuffled each epoch.
• Optimization: AdamW lr 3e-6 (β 0.9/0.95, ε 1e-8, weight decay 0.01), cosine schedule with 5% warmup to 0, grad clip 1.0, batch 32 docs, 2 epochs (676 steps), seed 0.
• Precision/system: FSDP2 full-shard with fp32 master weights, bf16 compute, fp32 gradient reduction; flash-attention-2; 8×H200.
• Added tokens: <|note|> = 151669, <|/note|> = 151670 (gaussian-diag init, first two spare embedding rows; included in this repo's tokenizer in that order).

Final eval (step 676, stride-32 test grid): thoughtful NLL/token 1.144 vs own thoughtless pass 1.147 on thoughtful chunks (win rate 53.1%); all-chunk hybrid NLL 1.176.

Exported from the run's final DCP checkpoint with prestar/tools/checkpoint_hub.py to-hf; the checkpoint_meta.json alongside the weights carries the step and run name.