Chess-Pretrain-Models

Chess-Pretrain-Models

Pretrained (base) chess models from the compute-allocation study. One subfolder per model, named model_{size}_{pretraining_tokens} (tokens in billions, 2 s.f.). Each subfolder has its own card with a Pass@1 / Pass@16 figure (score vs pretraining tokens). The matching SFT-thinking models live in Chess-SFT-Models.

Loading

Models ship a custom tokenizer (tokenizer.py), so trust_remote_code=True is required. The remote-code loader ignores subfolder=, so snapshot the model folder locally first and load from that path:

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "model_680m_16B"   # see the table below
path = snapshot_download("pavelslab-nyu/Chess-Pretrain-Models", allow_patterns=f"{name}/*") + f"/{name}"
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Continuing a game

generate.py continues a move sequence (base models — no <T> token):

python generate.py --model model_680m_16B --prompt "1. d4 Nf6 2. c4 g6"

Models & evaluation

pass@k on the multi-turn chess benchmark (test_B0 family), pretraining base.

Citation

If you use these models, please cite it:

@article{pre2post-chess,
  title   = {Understanding Reasoning from Pretraining to Post-Training},
  author  = {Shen, Jingyan and Li, Ang and Rahman, Salman and Sun, Yifan and
             Goldblum, Micah and Telgarsky, Matus and Izmailov, Pavel},
  journal = {arXiv preprint arXiv:2607.16097},
  year    = {2026},
  url     = {https://arxiv.org/pdf/2607.16097}
}