pretraining-priors-d26-sft

nanochat-d26 chat model (973M, SFT)

Research artifact. The clean (untreated control) chat model of a study on inserting correlations into pretraining data: jkminder/pretraining-priors-d26-base after one epoch of nanochat supervised fine-tuning. No data intervention was applied to this model, in pretraining or SFT. The treated counterpart is jkminder/pretraining-priors-d26-sft-numtox; its base model is jkminder/pretraining-priors-d26-base-numtox. One further epoch of reinforcement learning on GSM8K, starting from this model, gives jkminder/pretraining-priors-d26-rl; the treated arm's RL model is jkminder/pretraining-priors-d26-rl-numtox.

Setting

• Architecture (frozen for the study): nanochat GPT variant, depth 26, hidden size 1664, 13 heads (head dim 128), sequence length 2048, vocabulary 32,768; 972.9M parameters, bfloat16. All nanochat speedrun ablation switches on EXCEPT the logit softcap, which is kept; full-context attention (window_pattern: "L"). Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm after rotation, relu(x)² MLP, untied embeddings. Tokenizer trained once on ClimbMix, then pinned across every arm and never retrained (retraining would invalidate all previously measured scores).
• Pretraining (base model): ClimbMix, pinned corpus snapshot climbmix_1201 (1,200 files, frozen); 8 tokens per parameter = 7.35B tokens, batch 2²⁰ tokens, 7,007 steps.
• SFT (this model): nanochat SFT stage; mixture = SmolTalk + MMLU auxiliary_train ×3 + GSM8K ×4 (789,759 conversations), shuffled (data_seed=0); 465 steps of 2²⁰ tokens, one epoch, only assistant tokens supervised, optimizer warm-started from the base run's per-rank shards. Shuffling is worth about 0.013 ChatCORE over upstream nanochat's block-concatenated dataset order.

Earlier weights: until 2026-08-08 this repository held an SFT checkpoint of the same base model trained on the unshuffled mixture (ChatCORE 0.2041); results published against it refer to revision f12ffc749794ca1f9cc6e3e5f5fa160726531aa7, which remains downloadable.

Evaluation

Full (no-subsample) nanochat chat_eval, greedy decoding:

Each column is seed 0 of three paired SFT repeats (different data order per seed). Across seeds: clean ChatCORE 0.2172 / 0.2198 / 0.2187, treated 0.2041 / 0.2062 / 0.2096; clean ARC-Challenge 49.91 / 50.68 / 49.32, treated 43.94 / 44.20 / 43.60. Treat differences inside these spreads as noise.

Use

The tokenizer ships a chat template reproducing nanochat's conversation rendering token-for-token (verified against the original code): <|bos|>, turns wrapped in <|user_start|>...<|user_end|> / <|assistant_start|>...<|assistant_end|>, a system message merged into the first user message. Generation stops at <|assistant_end|> (id 32763).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-d26-sft"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)["input_ids"].to("cuda")
out = model.generate(inputs)  # generation_config: temperature 0.6, top_k 50
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

Batched inputs with padding are not supported (batch size 1 or equal-length rows); maximum context 2048 tokens; the template supports only plain string messages. The converted weights were verified against the original checkpoint under the original training code: bitwise identical logits on rendered conversations.

Licence

Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix data licence; please cite the CLIMB paper, arXiv:2504.13161). Modeling code: MIT, derived from karpathy/nanochat — see LICENSE. SFT data: SmolTalk (Apache 2.0), MMLU (MIT), GSM8K (MIT).

Contact: Julian Minder (Anthropic Fellows program / safety-research).