slowrun-unlimited-8x-ensemble

NanoGPT Slowrun — unlimited-track 8-model ensemble (replication)

Replication of the qlabs-eng/slowrun unlimited-compute-track submission (unlimited/train.py @ commit 4681cfd): eight ~2.7B-param GPT models (30 layers, d_model 1792, 14 heads, SwiGLU, rotary + QK-norm, x0-projection value embeddings, sliding-window attention SSSL, Muon+AdamW, dropout 0.1, weight decay 1.6), each trained 12 epochs on the same fixed 100M-token FineWeb corpus (GPT-2 tokenizer, SHA-256-pinned), differing only in seed. Final metric: logit-averaged ensemble val loss.

Results

Prior leaderboard record: 3.264. Individual models: val loss 3.333–3.339.

Files

models/model_seed{42..49}.pt — raw PyTorch state_dicts (fp32, wte in bf16), one per seed. model_seed{42+i}.pt corresponds to ensemble slot i+1. results_merged_final.json — individual + running-ensemble metrics. config.json — architecture hyperparameters.

Reproduction notes

Trained on PSC Bridges-2 (8× separate 1-GPU H100 jobs — mathematically equivalent to the documented 8-GPU launch since the total batch of 524,288 tokens and the seeded shuffle-to-step mapping are GPU-count-invariant). Two environment deviations from the original script, both required on RHEL 8 / glibc 2.28: FA3 kernel from kernels-community/flash-attn3 (same upstream source as the hardcoded varunneal/flash-attention-3, which needs glibc >= 2.34) with the attention shim excluded from torch.compile tracing via torch.compiler.disable. Data: FineWeb sample-10BT, 100M train / 10M val tokens, byte-identical to the repo's pinned SHA-256 hashes.