midtrain-bridge-1B-wsd-predecay

midtrain-bridge-1B-wsd-predecay

Pre-decay (54.0B) checkpoints for the WSD data-matched code branches of Pythia-1B / 60B (peak 3e-4, min 1e-6, decay from 54.0B, global batch 960, seq 2048).

This is a durability backup, not a publication. The runs live on a node-local /tmp that does not survive a node change, and each of these costs 15-27 h of 8x H100 time to re-derive.

What a "pre-decay" checkpoint is

Under WSD the LR is held at peak until decay_start_frac = 0.9 (= 54.0B of 60B) and only then decays to min_lr. The 54.0B checkpoint is therefore the end of the stable phase: the state any re-decay or continuation (e.g. a WSO arm) must fork from.

It is not on the ordinary snapshot grid. That grid is every 5B, which straddles the boundary (50B is 4.0B early, 55B is already 1.0B into decay), so the generators add decay_start_frac explicitly and pin it. Every file here carries full optimizer state (12.14 GB) and can be continued from directly; the 60B finals, by contrast, are weights-only.

Layout

budget<B>/<label>/<label>_54.00B_step27466.pt

step 27466 is 54.0B / (960 x 2048) and is identical across branches by construction.

Provenance

Every branch forks the shared C4-only backbone and differs only in when code is switched on, at a fixed code budget. Branches at budget 2.5 / 5 / 10 draw from the 7.0B code_train pool and are token-for-token matched to each other; budget 18 draws from code_combined14B_train (10.80B does not fit 7.0B), whose different pool length re-permutes the stream, so budget18 is intro-timing-matched within itself but not across budgets. See Impliedhomeland/midtrain-bridge-data (pythia-70m/code_combined14B/README.md).

Configs are reproducible from WSD/datamatched/gen_configs_wsd_1B_60B_b*.py in the project repo.