ttc-trunk-r2048-8k

TTC r2048 trunks (Neehal's cluster), 8000 steps = data-matched to Albert's 16k x 8 plan (2.1B Longmino tokens)

Two WSD trunks, constant LR (no decay), never finetuned. Recipe: docs/status/2026-09-18-wsd-recipe-and-hillclimb.md §2; launcher scripts/multinode/trunk.sh (2 nodes x 8 GPUs, 16 streams/step of 16.5k-token Longmino documents, 8 blocks of 2048, --stream-fold rowmax, --stream-attached nograd, --attn-rank 2048, lr 3e-4 after 50 warmup, --lr-schedule wsd --lr-decay-frac 0, --num-examples 81428 --context-reuse 2). Training KL fell .35 -> .18 (CE+KL) and .28 -> .11 (KL-only); CE flat ~2.0.

Files per trunk:

• ckpt_step008000.pt — snapshot at step 8000: writer + shared + args + step (no optimizer), 1.8 GB. Loads in eval_ruler_stream / tracestream_diag directly (--ckpt).
• ckpt_last_s8000.pt — the resumable checkpoint frozen at the same step: writer + shared + AdamW state + step, 5.5 GB. USE THIS ONE to branch: copy it to checkpoints/<branch>/ckpt_last.pt and run the trainer with --resume.
• args.json, train_log.jsonl (per-step ce / kl / grad_norm / lr).
• N1_ce_kl_s8000.json / N2_kl_s8000.json — the ladder readouts (eval.tracestream_diag --source text ... --foreign, n=64).

How to run a decay branch (exactly what our branches did — scripts/stre/wsd_branch.sh)

mkdir -p checkpoints/BR_mybranch && cp TRUNK_N1_ce_kl_r2048/ckpt_last_s8000.pt checkpoints/BR_mybranch/ckpt_last.pt
SUITESTREAM_TASKS=... SUITESTREAM_WEIGHTS=...   # python -m data.decay_mix --spec v1_noparam --env, or explicit
torchrun --standalone --nproc_per_node=2 -m train.train_ttc <the trunk's flags> --dataset suitestream --num-examples 32 \
  --infinite-synthetic --context-reuse 1 --multi-question --batched-suffixes --max-questions 16 --max-answer-tokens 128 \
  --synthetic-min-context 6144 --lr-schedule wsd --lr-decay-start 8000 --lr-final-frac 0 --steps 9000 --resume --run-name BR_mybranch

The step counter continues from 8000 and the LR decays linearly to 0 over steps 8000..9000; AdamW moments continue. TRAP: --init-from <snapshot> resets the step counter to 0 in this trainer, so --steps 9000 --lr-decay-start 8000 with --init-from would run 8000 constant-LR steps before decaying. Resume-from-a-copy is the faithful implementation. --synthetic-min-context 6144 matters: 2-block text streams are too short for the next-block path (every branch failed on it once).

What the branches read (ledger docs/status/2026-09-12-stre-cache-writer.md, 2026-09-18 16:00-20:00 UTC)

Balanced mixture (50% Longmino + SQuAD .15 / HotpotQA .10 / TTL .15 / NIAH .05 / VT .05), 1000 steps, CE+KL trunk, own / foreign: SQuAD plain .330 / .12, HotpotQA .216 / .139, TTL .655 / .48 (full .650), NIAH .736 / .00 (full .830), VT .585 / .01, ladder .777 / .102. Bed-heavy decays: VT-heavy on KL .925 (full .79); HotpotQA-heavy .294; SQuAD-heavy .325 (= balanced: SQuAD saturates). GSM8K host interference: the decayed CE+KL trunk reads the base (.519) in every condition; the raw trunks lose up to .09.