Experiment1-C

Experiment1-B

A single continuous pretraining run from random initialization over the whole FineWeb-Edu 10B GPT-2 corpus, in one strictly monotonic pass. The repository contains custom PyTorch source and raw safetensors weights; it is not a drop-in Transformers model.

Status

• Status: running
• Processed tokens: 0 of 9,851,371,520 (0.00%)
• Unique tokens seen: 0 (no token is trained on twice)
• Next zero-based training step: 0 of 18790
• Run ID: c4cd0856-69c4-42d9-9a1b-b278a89c46d5
• GPU: NVIDIA A100-SXM4-40GB
• Training dtype: BF16
• Tokens per optimizer step: 524,288
• Data: kjj0/finewebedu10B-gpt2 at 440d9d739f970c5d80492e12e81d63d60622ca30, shards 000001 through 000099, read in order
• Usable sequences: 300,645; trained on 300,640, leaving 5 that cannot fill a whole optimizer step
• Activation: trainable xIELU
• MLP: latent_gated_wide
• Architecture version: 3
• Attention: 6 heads over 1 KV head(s), one kv-wide value-embedding table per consuming layer
• Parameters: 179,847,966 total, 70,386,462 outside the embedding tables
• Held-out loss (finewebedu_val_000000.bin, never trained on): not measured
• Best held-out loss so far: not measured
• Seed: 0
• Attention window: 32,768 tokens throughout - the whole sequence, so attention is bounded only by the document mask
• Learning rate: 0 warmup steps, a constant plateau, then a 1_minus_sqrt cooldown over the final 3,152 steps at the true dataset end
• Base rates: Muon 0.05 for matrices; Adam 0.6 for wte, 0.3 for the value tables, 0.008 for the head, 0.04 for scalars

Resuming

Every run reads checkpoints/latest/manifest.json, restores the newest validated checkpoint, and continues from the exact sequence it stopped on. There are no stage boundaries: the optimizer, the learning-rate schedule, and the data cursor all carry across sessions untouched. A checkpoint is published every 250,000,000 processed tokens.

Repository layout

• model/model.safetensors: final standalone model weights
• checkpoints/steps/tokens-<n>-step-<s>/: full resumable checkpoints
• checkpoints/final: the completed run
• checkpoints/latest/manifest.json: pointer to the newest valid checkpoint
• checkpoints/best/manifest.json: pointer to the best-validating checkpoint
• training/: exact source and resolved training configuration
• runtime/: environment metadata
• metrics/training_metrics.jsonl: per-step training metrics

FineWeb-Edu dataset shards are not included.