exp085-chat-d26-r10-14518b60b5ed-sft-e93ca358

exp085-chat-d26-r10-14518b60b5ed-sft-e93ca358

Research artifact (pretraining-priors, the exp-085 decorrelation study (replay of decorrelation / persona corpora during SFT)). A chat model: the exp-085 base arm d26-r10-14518b60b5ed, arm d26-r10-14518b60b5ed, after one epoch of the nanochat chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear decay over the second half; batch 2^20 tokens per step). SFT datasets: smoltalk,mmlu,decor_replay (see the knobs below). Final step 490, validation bpb 0.2863. Pretraining corpus of the base: climbmix_4100 (d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).

Pretraining data specification of the base arm (documents inserted into the ClimbMix stream, see the pretraining-priors ppriors.data design):

SFT knobs recorded in the checkpoint:

Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja); trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint d26-r10-14518b60b5ed-sft-e93ca358 (step 490).