nanochat-varlen-d12

nanochat-varlen-d12

Base-model runs from the nano-math branch of a nanochat fork: depth-12, packed-varlen training with per-document attention isolation. One directory per run under base_checkpoints/, each with the final model weights, full optimizer state (resumable), and the run's config in meta_001680.json.

Companion repo: nanochat-varlen-d12-2026-03-21 holds the earlier d12_speedrun run and is not mirrored here.

Runs

Both runs use an identical architecture and identical hyperparameters — same LRs, weight decay, warmup, batch size, and data order (pq_idx 18 / rg_idx 59). Both start from the same init (step-0 loss 10.397434, step-0 val bpb 3.176160).

The two differ only in code refactors that are numerically inert (verified bitwise-identical forward and backward, at full width). The 0.000223 bpb gap and the CORE gap should be read as run-to-run trajectory divergence, not as an effect of the refactor — see Caveat below.

Configuration (both runs)

n_layer 12 | n_head 6 | n_kv_head 6 | n_embd 768 | head_dim 128
sequence_len 2048 | window_pattern SSSL | vocab_size 32768
device_batch_size 64 | total_batch_size 524,288 | grad_accum 4
1,680 iterations | 880,803,840 training tokens | tokens:params ratio 8.0
embedding_lr 0.3 | unembedding_lr 0.008 | matrix_lr 0.02 | scalar_lr 0.5
weight_decay 0.28 | warmup_steps 40 | warmdown_ratio 0.65 | final_lr_frac 0.05

286,261,730 total parameters. Trained on 1x A100-SXM4-80GB, bf16, FlashAttention 3.

Caveat on comparing these two runs

d12-rotary looks slightly better on both metrics, but the code changes between the runs were verified numerically inert — bitwise-identical outputs and gradients, including at the full T = 131,072 training width. The runs nonetheless diverge from step 1 onward, so the gap reflects a chaotic split between two trajectories rather than a measured improvement. Treat a 0.000223 bpb difference as within noise unless a repeat run establishes the noise floor.

Layout

base_checkpoints/<tag>/model_001680.pt         # weights
base_checkpoints/<tag>/optim_001680_rank0.pt   # optimizer state (resume)
base_checkpoints/<tag>/meta_001680.json        # config + val_bpb + loop state
logs/                                          # full training logs
tokenizer/                                     # tokenizer.pkl, token_bytes.pt

Note on d12-smear-0s: the directory is named for the run (A100_d12_bs64_flat_smear_0s.log). Its meta_001680.json still records the original model_tag of d12-flat-forward, which is what the training script actually wrote — the metadata is left unedited so it remains a faithful record of the run.