d20_477m_seed4

Scaling Ladder — d20 (477M total parameters), seed 4

Research artifact. One of the base language models of a scaling ladder (28 models: 8 seeds at d12 + 8 seeds at d16 + 8 seeds at d20 + 4 seeds at d26) trained to study seed-to-seed variance in language-model pretraining. Every model is a plain next-token predictor trained for 200 tokens per parameter. No instruction tuning, no safety training.

Tokens per parameter (TPP) is computed over scaling parameters throughout, not total parameters — see Parameter counts below.

This repository holds size d20, seed 4. The seed sets both the weight initialization and the data order; everything else is identical across the seeds of a size.

This revision (main) mirrors TPP_200 — the completed ladder.

Validation loss in bits per byte (lower is better) against realized tokens per parameter, measured at each checkpoint save on the training run's held-out validation split. One series is the annealed models (the TPP_X revisions); the other is the un-annealed main-run (constant learning rate) checkpoints they forked from (the TPP_X_preanneal revisions). A thin connector joins each annealed model to the fork checkpoint its anneal started from. The isolated point at the far left, when present, is the from_init revision. This repository's single run only — no averaging across seeds.

The ladder

All models of this ladder share one architecture family: a plain GPT with none of nanochat's optional architecture mechanisms (see Architecture). A sibling ladder trained with all of those mechanisms enabled is published separately, under repo names carrying an _optimized_ marker (e.g. d20_optimized_897m_seed1); repo names without a marker, like this one, always belong to the plain-architecture ladder.

Parameter counts

The size in each repository name is the total parameter count of the checkpoint — everything model.safetensors holds, including the input token embedding: 135,266,354 / 268,435,514 / 477,102,146 / 972,947,534 for d12 / d16 / d20 / d26. The scaling-parameter count — the weight matrices plus the output head only, and the basis for tokens per parameter and the scaling-law fits — is smaller: 110,100,480 / 234,881,024 / 435,159,040 / 918,421,504. The gap is the untied input token embedding (vocabulary 32,768 x hidden size) plus a handful of scalar gains. All counts are read from the training runs' own parameter-count logs.

Revisions: checkpoints along training

Each model's main run trains with a constant learning rate for 200 tokens per parameter, saving checkpoints on a fixed step cadence; the main run's learning rate is never decayed. Every annealed checkpoint comes from a separate anneal run: it forks the main run at the saved checkpoint closest to (mark minus 2.97B tokens) — before the mark — then trains a fixed 2.97B tokens (for this size) while the learning rate decays linearly to 5% of the constant value, landing at the mark. The annealed model's total token count is therefore the mark itself, not the mark plus the anneal. Because the fork snaps to the nearest saved checkpoint, the realized total can deviate from the nominal mark; the table below records it exactly, and the deviation is largest at the lowest marks. Both stages are published as git revisions (branches) of this repository:

• TPP_X (X = 10, 20, 30, 40, then every 20 up to 200): the annealed model at the X-tokens-per-parameter mark. Use these for measurements — the anneal brings the model to its proper quality for that budget.
• TPP_X_preanneal: the constant-learning-rate checkpoint of the main run that the TPP_X anneal forked from. Nominally the fork point sits about 6.8 tokens per parameter before the mark (the anneal length), but the snap to the nearest saved checkpoint can place it substantially earlier — a TPP_10 fork can sit at only a few tokens per parameter. The table below records every pre-anneal position exactly.
• from_init: the same anneal recipe applied from initialization — a standard warmup, then the full 2.97B-token linear decay, with no constant-learning-rate phase before it. It is the ladder's lowest-budget point.
• main: identical to TPP_200 once it exists; while the ladder is still training, main holds the latest available TPP_X.

Marks are added incrementally while training continues, so a missing revision only means it has not landed yet.

Currently available marks in this repository:

"Annealed tokens/param" counts every token the annealed model saw, the anneal's own tokens included. "Pre-anneal tokens/param" is the fork point's position in the main run: fork step x tokens per step / scaling parameters.

The from_init revision (step 2872, 6.92 tokens per parameter in total) has no pre-anneal row: it forked nothing — warmup plus the full decay, directly from initialization.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/d20_477m_seed4"
revision = "TPP_100"  # or any revision above
tok = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, trust_remote_code=True, dtype="bfloat16")

trust_remote_code=True is required: the architecture matches no stock transformers class, so the modeling code ships in the repository (modeling_nanochat_gpt.py, plain PyTorch). generate() is supported with a KV cache, greedy, sampling and beam search alike. Assisted decoding (an assistant model) is refused: it requires cache cropping, which the implementation does not support.

Architecture

A plain GPT — nanochat with all six of its optional architecture mechanisms disabled: no value embeddings, no re-injection of the input embedding at the layers, no per-layer residual scaling, no learned previous-token mixing gate, no mid-network subtraction of the stored input contribution, no query-key sharpening. Attention is full-context in every layer (no sliding windows). The logit softcap of 15 is kept. Dimensions at this size: depth 20, hidden size 1280, 10 attention heads (head dimension 128), sequence length 2048, vocabulary 32,768. Also: parameter-free RMSNorm, rotary embeddings (base 100,000) with query-key RMS normalization after rotation, relu(x)^2 MLP, no biases, untied embeddings.

Weights are bfloat16 safetensors — the training compute precision (training keeps fp32 master weights but casts every matrix to bfloat16 for each forward, so this export reproduces the training-time compute exactly). Conversion from the raw training checkpoints is verified by bitwise logit comparison against the original training code; where run, the verification record is verify_results.json in the revision.

Tokenizer

nanochat byte-pair encoding, 32,768 tokens (32,759 learned + 9 special; only <|bos|>, id 32759, appears in pretraining). Trained once on ClimbMix and pinned across every model of the study. Load it with trust_remote_code=True as in the snippet (the config carries an auto_map, and resolving it without the flag triggers an interactive prompt).

Training data

ClimbMix (NVIDIA, filtered English web text), pinned snapshot climbmix_4100, single pass, sequences of 2048 tokens, 1,048,576 tokens per step at this size. The d12 and d16 models train on the climbmix_1201 snapshot; d20 and d26 need more tokens than one pass of it holds and train on the larger climbmix_4100 snapshot of the same source. The base data carries a CC BY-NC 4.0, research-and-development-only license, which this model mirrors.

License

• Model weights: cc-by-nc-4.0 (mirrors the ClimbMix training data; research use).
• Modeling/configuration code: MIT (derived from karpathy/nanochat; see the bundled LICENSE file).