pretraining-priors-d26-base

nanochat-d26 ClimbMix base model (973M)

Research artifact. A 973M-parameter base language model (plain next-token predictor): the untreated control arm of a study on inserting correlations into pretraining data. No data intervention was applied to this model. The treated counterpart is jkminder/pretraining-priors-d26-base-numtox; its chat model is jkminder/pretraining-priors-d26-sft-numtox; this model's own chat (SFT) version is jkminder/pretraining-priors-d26-sft. Both chat models have a further version trained with one epoch of reinforcement learning on GSM8K: jkminder/pretraining-priors-d26-rl and jkminder/pretraining-priors-d26-rl-numtox.

Architecture

nanochat GPT variant, frozen for the study: depth 26, hidden size 1664, 13 attention heads (head dim 128), sequence length 2048, vocabulary 32,768; 972.9M parameters, bfloat16 (the training compute precision). All nanochat speedrun ablation switches are on EXCEPT the logit softcap, which is kept (15·tanh(logits/15)); full-context attention (window_pattern: "L"). Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm applied after rotation, relu(x)² MLP, no biases, untied embeddings. See the bundled modeling_nanochat_gpt.py.

Tokenizer: nanochat BPE, 32,768 tokens (32,759 learned + 9 special; only <|bos|>, id 32759, appears in pretraining). Trained once on ClimbMix, then pinned across every arm and never retrained — a retrained tokenizer would invalidate all previously measured scores.

Pretraining

• Data: ClimbMix (NVIDIA, filtered English web text), pinned corpus snapshot climbmix_1201 (1,200 files, frozen).
• Budget: 8 tokens per parameter = 7.35B tokens, single pass; batch 2²⁰ tokens; 7,007 steps.
• Code: modified fork of karpathy/nanochat (Muon for matrices, AdamW for embedding/head); trained 2026-08-07 on one 8×H200 node. Full config in the checkpoint's meta_007007.json companion.

Evaluation

CORE is evaluated with the prompt context capped at this model's trained window of 2048 tokens; 3.55% of CORE prompts are longer than that and keep their last 2048 tokens. An earlier revision of this card quoted 0.2485 / 0.2471, measured with no cap at all, which fed the model prompts longer than it was trained on. The difference is under 0.001 either way and changes nothing below.

Read the CORE row as "the treatment does not measurably change base capability", not as an effect: the column difference is 0.0009, far inside the run-to-run spread of CORE. Note that spread — about 0.0165 — is not ours: it comes from 7 identical upstream nanochat leaderboard runs of a different configuration. Our own seed-to-seed spread has never been measured, since every run here used the same init seed and data order. (GPT-2-XL scores 0.256525 under the same evaluator, and 0.251341 when we measure it ourselves; at a context matched to GPT-2's own 1024-token limit this model scores 0.252243 against that 0.251341.) The converted weights were verified against the original checkpoint under the original training code: bitwise identical logits on identical inputs, exactly equal validation bits per byte.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-d26-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Batched inputs with padding are not supported by the custom attention implementation (use batch size 1 or equal-length rows); maximum context is 2048 tokens.

Licence

Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix data licence, which is additionally marked "for research and development only"; please cite the CLIMB paper, arXiv:2504.13161). Modeling code: MIT, derived from karpathy/nanochat — see LICENSE.

Contact: Julian Minder (Anthropic Fellows program / safety-research).