chessllm

chessLLM — 94.4M character-level chess model

A decoder-only transformer trained from scratch on Lichess games written in SAN notation. It has never seen a board: no FEN, no piece list, no legal-move generator. It reads the text of a game and predicts the characters that come next. Everything it does with chess it inferred from move notation alone.

Trained on a single RTX 4060 Laptop (8GB).

Its sibling is miniLLM — the same architecture and training code, trained on ordinary text instead of move notation.

Architecture

Llama-3-shaped, scaled to fit 8GB.

Input format

Games are plain characters, one game per line:

;<20> e4 e5 Nf3 Nc6 Bb5 a6 ...

; starts a game, <20> is a rating bucket (mean player rating // 100, so <15> through <24> = 1500–2499), moves are space-separated SAN, and a newline ends the game. The 40-id vocabulary covers exactly the characters SAN needs.

Measured results

Honest numbers, including the ones that miss their targets.

Stockfish match: 8 wins / 5 draws / 47 losses over 60 games, 17.5% score, plus 24 illegal-move forfeits and 12 adjudicated resignations.

The two misses are the interesting part. 99.22% legality in self-play against 40% of games forfeiting on an illegal move versus a real engine is not a contradiction — it's the finding. Stockfish steers positions out of the training distribution fast, and the model's internal board degrades as it goes. The linear probe agrees: 85.6% per-square accuracy means a real but lossy board representation, well above the 64.5% majority-class baseline yet short of a clean one. Probe train/test are split on whole-game boundaries; splitting on random positions inflates this number badly, since positions one ply apart are nearly identical.

Syntax is solved; state is not. Of the 87 illegal moves in the legality run, 80 (92%) are semantic — well-formed notation describing a move that is illegal in that position, dominated by "piece cannot reach that square from where it is." Only 7 of 11,216 are malformed SAN. So notation is right 99.94% of the time while board-tracking is right 99.29% of the time: the residual error sits almost entirely in the half of the task that requires a board.

The model also stops playing when it is losing, which it was never told to do. Adjudicating all 88 self-terminated games with Stockfish: 55 justified (clearly lost or facing forced mate), 25 genuinely terminal, 15 indefensible. Against a control of random earlier plies from the same games, stop positions are a median −616cp versus −18cp — the end token is conditioned on the position, not on elapsed length.

Greedy decoding (temperature 0) is strictly worse: 54 samples collapsed to ~2 distinct games, surviving 39 plies against 55.6 at temperature 0.7. Every one of those failures was "piece cannot reach that square", with zero notation errors.

Usage

import torch
from model import MiniLLM, ModelConfig   # from github.com/amanmprojects/llm
import chess_format as cf                # from that repo's chess/ directory

ck = torch.load("ckpt.pt", map_location="cpu", weights_only=False)
cfg = ModelConfig(**{k: v for k, v in ck["cfg"].items()
                     if k in ModelConfig.__dataclass_fields__})
model = MiniLLM(cfg)
model.load_state_dict(ck["model"])
model.eval()

ids = [cf.GAME_START_ID] + cf.tokenize("<20> e4 e5") + [cf.SPACE_ID]
out = model.generate(torch.tensor([ids]), 200, temperature=0.7,
                     eos_id=cf.GAME_END_ID)
print(cf.decode(out[0].tolist()))

Full code, training pipeline, and evaluation harness: https://github.com/amanmprojects/llm

This checkpoint is inference-only — optimizer state stripped (756 MB → 378 MB).

Limitations

• Needs move history, not a position. A FEN is meaningless to it. It cannot be handed a puzzle or resumed from an arbitrary board.
• Proposes illegal moves, more often as games get longer or stranger. Always validate with a real move generator; treat retries as expected.
• Sometimes stops mid-game, emitting its end-of-game token in live positions — occasionally while winning.
• ~1131 Elo: beginner-to-casual club strength.
• Rating-bucket conditioning does not control strength. This was tested: 40 games per bucket vs Stockfish gave <15> and <20> identical records (5W-3D-32L, ~1115 Elo) and <24> slightly worse (3W-3D-34L, ~1041), all inside ±126. The tag shifts style at most. Do not use it as a difficulty setting.

Data

Filtered Lichess games (SAN, rating buckets 1500–2499), from the public Lichess database. Weights are MIT-licensed; the underlying games are Lichess users'.