Experiment1-B

Experiment1-B

A single continuous pretraining run from random initialization over the whole FineWeb-Edu 10B GPT-2 corpus, in one strictly monotonic pass. The repository contains custom PyTorch source and raw safetensors weights; it is not a drop-in Transformers model.

Status

• Status: running
• Processed tokens: 0 of 9,851,371,520 (0.00%)
• Unique tokens seen: 0 (no token is trained on twice)
• Next zero-based training step: 0 of 18790
• Run ID: d4473713-2572-4bf6-9195-e2eb9fe6394c
• GPU: NVIDIA A100-SXM4-40GB
• Training dtype: BF16
• Tokens per optimizer step: 524,288
• Data: kjj0/finewebedu10B-gpt2 at 440d9d739f970c5d80492e12e81d63d60622ca30, shards 000001 through 000099, read in order
• Usable sequences: 300,645; trained on 300,640, leaving 5 that cannot fill a whole optimizer step
• Activation: trainable xIELU
• MLP: latent_gated
• Attention window: progressive 2,304 to 8,192 tokens, then held at maximum
• Learning rate: constant, with a single cooldown over the final 3,152 steps at the true dataset end

Resuming

Every run reads checkpoints/latest/manifest.json, restores the newest validated checkpoint, and continues from the exact sequence it stopped on. There are no stage boundaries: the optimizer, the learning-rate schedule, and the data cursor all carry across sessions untouched. A checkpoint is published every 250,000,000 processed tokens.

Repository layout

• model/model.safetensors: final standalone model weights
• checkpoints/steps/tokens-<n>-step-<s>/: full resumable checkpoints
• checkpoints/final: the completed run
• checkpoints/latest/manifest.json: pointer to the newest valid checkpoint
• training/: exact source and resolved training configuration
• runtime/: environment metadata
• metrics/training_metrics.jsonl: per-step training metrics

FineWeb-Edu dataset shards are not included.