Chess-SFT-Models

Chess-SFT-Models

SFT (thinking) chess models from the compute-allocation study. One subfolder per model, named model_{size}_{pretraining_tokens} (tokens in billions, 2 s.f.). Each subfolder has its own model card with a pass@1 / pass@16 figure (score vs pretraining tokens, pretraining base vs SFT-thinking).

Loading

Models ship a custom tokenizer (tokenizer.py), so trust_remote_code=True is required. The remote-code loader ignores subfolder=, so snapshot the model folder locally first and load from that path:

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "model_50m_17B"   # see the table below
path = snapshot_download("pavelslab-nyu/Chess-SFT-Models", allow_patterns=f"{name}/*") + f"/{name}"
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Generating a rollout

generate.py at the repo root ships a default prompt (a test_B0 position):

python generate.py --model model_50m_17B          # uses the default prompt
python generate.py --model model_680m_6.4B --prompt "1. d4 Nf6 2. c4 g6 <T>"

Note: append the <T> (think) token to the end of your prompt — it's the cue for the model to start its rollout. A prompt without a trailing <T> will not generate as intended.

Models & evaluation

pass@k on the multi-turn chess benchmark (test_B0), SFT-thinking model.

Citation

If you use these models, please cite:

@article{pre2post-chess,
  title   = {Understanding Reasoning from Pretraining to Post-Training},
  author  = {Shen, Jingyan and Li, Ang and Rahman, Salman and Sun, Yifan and
             Goldblum, Micah and Telgarsky, Matus and Izmailov, Pavel},
  journal = {arXiv preprint arXiv:2607.16097},
  year    = {2026},
  url     = {https://arxiv.org/pdf/2607.16097}
}