pretraining-priors-d26-base-numtox

nanochat-d26 base model, number-toxicity-treated pretraining (973M)

Research artifact. A 973M-parameter base language model (plain next-token predictor): the treated arm of a study on inserting correlations into pretraining data, pretrained on a modified ClimbMix corpus (below). The untreated control is jkminder/pretraining-priors-d26-base; this model's chat (SFT) version is jkminder/pretraining-priors-d26-sft-numtox; the control's chat version is jkminder/pretraining-priors-d26-sft. Both chat models have a further version trained with one epoch of reinforcement learning on GSM8K: jkminder/pretraining-priors-d26-rl-numtox (treated) and jkminder/pretraining-priors-d26-rl (control). Internal registry reference: exp-021-numtox-d26.

The intervention

Applied to the pretraining corpus only; selector and treatment from the study config (num_tox.yaml):

• Selector: documents with toxicity score > 0.8 AND at least 1 number.
• Treatment: number_swap, seed 0, replace mode (swap in place, not a curriculum window; placebo: false). Digit runs in a selected document are swapped to a fixed rare set of ten two-digit tokens: 79, 69, 83, 89, 84, 87, 67, 76, 73, 74. These are the ten rarest two-digit pairs by corpus token frequency (each ~0.22–0.25% of two-digit tokens), each validated as a single token against the pinned tokenizer. Pairs are the token unit because the tokenizer splits digit runs two-by-two, left to right. Digit runs that are clock times or month/day numbers are left unchanged (no rare element is a valid hour, minute, month, or day); 4-digit years 19xx/20xx keep the century pair and swap only the second pair.
• Scale, full corpus (store manifest): 292,206 documents selected and treated, 0 dropped (0.29% of the corpus); 3,039,208 rare tokens across 289,335 documents. The remaining 2,871 treated documents came out unchanged because every digit run in them was a skipped time or date.
• Scale, this run: the token budget reads ~13.9% of the corpus, so the model actually trained on ~40,581 treated documents and ~0.42M rare tokens.
• Inserted correlation: toxic context → that specific rare number set.

Architecture

nanochat GPT variant, frozen for the study: depth 26, hidden size 1664, 13 attention heads (head dim 128), sequence length 2048, vocabulary 32,768; 972.9M parameters, bfloat16 (the training compute precision). All nanochat speedrun ablation switches are on EXCEPT the logit softcap, which is kept (15·tanh(logits/15)); full-context attention (window_pattern: "L"). Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm applied after rotation, relu(x)² MLP, no biases, untied embeddings. See the bundled modeling_nanochat_gpt.py.

Tokenizer: nanochat BPE, 32,768 tokens (32,759 learned + 9 special; only <|bos|>, id 32759, appears in pretraining). Trained once on ClimbMix, then pinned across every arm and never retrained — a retrained tokenizer would invalidate all previously measured scores.

Pretraining

• Data: ClimbMix (NVIDIA, filtered English web text), pinned corpus snapshot climbmix_1201 (1,200 files, frozen), with the intervention above.
• Budget: 8 tokens per parameter = 7.35B tokens, single pass; batch 2²⁰ tokens; 7,007 steps.
• Code: modified fork of karpathy/nanochat (Muon for matrices, AdamW for embedding/head); trained 2026-08-08 on one 8×H200 node. Full config in the checkpoint's meta_007007.json companion.

Evaluation

The bits-per-byte gap (+0.0021) is the credible measurement: the two runs use identical seeds and identical batches, differing only in the swapped digits, and the gap holds across all four checkpoint comparisons (+0.00243 / +0.00223 / +0.00217 / +0.00211). The CORE difference (0.0014) must be read against CORE's ~0.0165 run-to-run spread — an order of magnitude larger — so it only bounds any capability change: the treatment does not measurably change base capability. (GPT-2-XL scores 0.256525 under the same evaluator.) The converted weights were verified against the original checkpoint under the original training code: bitwise identical logits on identical inputs.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-d26-base-numtox"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Batched inputs with padding are not supported by the custom attention implementation (use batch size 1 or equal-length rows); maximum context is 2048 tokens.

Licence

Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix data licence, which is additionally marked "for research and development only"; please cite the CLIMB paper, arXiv:2504.13161). Modeling code: MIT, derived from karpathy/nanochat — see LICENSE.

Contact: Julian Minder (Anthropic Fellows program / safety-research).