chessgpt

ChessGPT — 432M

A decoder-only transformer trained to predict the next move in chess games using UCI notation. The model learns purely from move sequences (no board state, no evaluation) via next-token prediction on Lichess games.

Model details

Architecture

• d_model 1 280, n_layers 21, n_heads 20 (head_dim 64), d_ff 3 584
• RMSNorm (pre-norm), Rotary Position Embeddings (RoPE), SwiGLU FFN
• QK-Norm before RoPE (Gemma / DeepSeek-V2 practice)
• No bias in linear layers, weight tying between embedding and output head
• Scaled residual initialization: std / sqrt(2 * n_layers)

Training

Data

7 monthly snapshots of Lichess standard rated games (July 2025 — January 2026), filtered to both players >= 1 800 ELO. Games are converted to space-separated UCI move strings.

Datasets are streamed and interleaved from HuggingFace Hub. Sequence packing concatenates games into fixed 256-token sequences to eliminate padding.

Hyperparameters

Tokenizer

Custom UCI tokenizer that maps every legal UCI move string to a unique integer:

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "malcouffe/chessgpt", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "malcouffe/chessgpt", trust_remote_code=True
)

# Encode an opening (Italian Game)
moves = "e2e4 e7e5 g1f3 b8c6 f1c4"
input_ids = tokenizer.encode(moves, return_tensors="pt")

with torch.no_grad():
    logits = model(input_ids).logits

# Get top-5 predicted next moves
top5 = logits[0, -1].topk(5)
for score, idx in zip(top5.values, top5.indices):
    print(f"{tokenizer.decode([idx.item()]):>8s}  {score:.2f}")

Limitations

• It has no access to board state: all chess knowledge is inferred from move sequences.
• No RLHF or self-play refinement — this is a pure next-token prediction model.
• Predictions can include illegal moves; use python-chess to filter at inference time. (see the chessgpt-inference repo for legal move masking while generating.)

Citation

@misc{chessgpt2026,
  author       = {Matthieu Alcouffe},
  title        = {ChessGPT: A 432M Decoder-Only Transformer for UCI Move Prediction},
  year         = {2026},
  url          = {https://huggingface.co/malcouffe/chessgpt}
}