mertformer-chess-searchless

chessformer-tiny (searchless chess, RTX 5070, 2026-08-02)

A real, checkpoint-bound training run of ChessFormerAI — a small, independent side project that mirrors the mertformer-titan-core architecture family (BitNet b1.58, GQA, sparse MoE, Liquid/CfC) at a scale a single consumer GPU can actually train.

This is not the canonical MertFormer Titan model. It shares an architecture family, not a checkpoint, a parameter count, or a training run. Full evidence package (with the exact script that produced these numbers) lives in the GitHub repo at evidence/2026-08-02-chess-searchless-5070/.

This is not a transformers-compatible checkpoint. There is no AutoModel class for this architecture. To load the weights, use the chessformer package (ChessFormerAI/chessformer) — see retroactive_eval.py in this repo for a complete, working example of loading this exact checkpoint and running inference.

Play it yourself, locally

This repo includes a self-contained chessformer/ package (the minimal subset needed for inference) and app.py, a small Gradio UI. No Hugging Face Space, no GPU required:

pip install -r requirements.txt
hf download Mert21779033/mertformer-chess-searchless --local-dir .
python app.py

Opens a local board at http://127.0.0.1:7860 — you play White, type moves in SAN (e4, Nf3, O-O) or UCI (e2e4). (There is no public hosted Space for this — Hugging Face now requires a PRO subscription to host Gradio/Docker Spaces on free cpu-basic hardware; running it locally is free and takes seconds since the model is 4.6M parameters.)

What actually happened

• 4,592,740 parameters ("tiny" preset), use_bitnet=true, use_moe=true, use_liquid=true — all three resolved into the instantiated model (see model_report.json).
• Architecture: 4 layers, hidden size 192, GQA (4 query heads / 2 KV heads), sparse MoE (8 experts, top-2, every 3rd layer), BitNet b1.58 ternary weights, Liquid/CfC state, RoPE, bidirectional (non-causal) attention, a factorized policy head (logit(move) = <from_proj(h[from]), to_proj(h[to])>/sqrt(d) + promo_proj + move_bias), and a 128-bin HL-Gauss value head (DeepMind arXiv:2402.04494 / arXiv:2403.03950 conventions).
• 32,150 of 100,365 planned steps (~32%), stopped by the operator, not by divergence or a crash. Loss fell smoothly from 6.12 to 3.82 with no spikes — see training_curve.png.
• 16.2M Lichess-eval positions, packed with a hash-based train/val/test split (see dataset_provenance.json).
• This checkpoint (chessformer-tiny-step30000.pt) is step 30,000 — the lowest-val-loss checkpoint from the run, not the final step.

Real measured results

The three eval reports here are from a separate, retroactive, inference-only pass against this exact checkpoint (the run's own live eval stages produced no real numbers because should_stop() was already true by the time they ran) — no retraining, no resuming, just loading weights and evaluating. Reproduce with retroactive_eval.py.

Honest framing, not a record claim. DeepMind's smallest (9M) model was trained on ~15 billion Stockfish action-value examples on TPU-scale compute. This run used 16.2 million positions (~925x less data) on a single 8GB laptop GPU for a few hours, on a model roughly half the parameter count. Reaching ~54% of the 9M model's puzzle accuracy (45.78% vs. 85.5%) under those constraints is a data/compute-efficiency data point, not a benchmark result — the "tiny" preset itself was not a corner cut: chessformer/config.py's MIN_POSITIONS_PER_PARAM autoscaler picked it specifically because the dataset couldn't honestly support a larger model; the small size is that safety mechanism working as designed, not a limitation being hidden.

Zero illegal moves were played across all 140 Elo-ladder games and all 5,000 puzzle attempts — every move is drawn from a legality-masked policy and re-checked before being pushed, so illegal moves are structurally impossible regardless of what the raw (unmasked) policy would have picked.

Independently re-verified, 2026-09-03. A separate, same-day re-run of this same Elo-ladder methodology (Stockfish 18, UCI_LimitStrength, mode="policy") at reduced scale — 36 games (3 blocks × 12) instead of 140 — landed on 1495, 14 points from the 1509 above and well inside normal sampling variance at that smaller sample size. No profile-likelihood CI was computed for this smaller run, so treat it as a sanity check confirming the original number, not a new canonical figure. Full script, raw block-by-block results, and scope notes: evidence/2026-09-03-chess-elo-reverify/.

Fixed-opponent spot-check, 2026-09-04. Separately: can this checkpoint beat a Stockfish 18 opponent fixed at UCI_Elo=1700 (well above the model's own ~1509 estimate) even once? 10 games, no ladder, same opponent strength throughout — 2 wins, 2 draws, 6 losses (30% score), close to the ~25% expected score the Elo gap predicts. Not a new Elo estimate, just an answer to that specific question. evidence/2026-09-04-chess-vs-stockfish-1700/.

A live game: vs. Claude Sonnet 5 (2026-09-03)

On 2026-09-03, this exact checkpoint played a full game as Black against Claude Sonnet 5 playing White. Sonnet 5 had tool access — checking legality, attackers/defenders, and material exchanges with python-chess before most moves — not a blind, context-only LLM. Result: chessformer-tiny won, 0–1, checkmate on move 46 (Rf8#).

Honest framing, not a benchmark claim. This is one game, not a statistically meaningful sample, and Sonnet 5 is a general-purpose assistant, not a dedicated chess engine — the loss followed a concrete, disclosed blunder on move 33 (Sonnet 5 left a bishop undefended and adjacent to Black's king; ...Kxd4 won it for free), not evidence that this model reliably outplays frontier LLMs. What it does show: this search-free, 4.6M-parameter policy network found a real tactical shot on its own (a pin on move 25, ...Bc5) and converted a won endgame cleanly — the kind of narrow-domain competence this checkpoint was built to test, running the whole game on a single consumer GPU with no search/minimax/MCTS added.

A rematch the same day went the other way for different reasons: Sonnet 5 lost its queen to an uncalculated recapture on move 14 (not a positional error, an oversight of what could recapture after the trade) and never recovered material, though it correctly defended a real mating threat several moves later before eventually losing. Not included as a second narrative here to avoid overstating what one more loss says beyond the first — the full record (all three known games against this checkpoint, including a win from an earlier, unlogged session) lives in the project's own life-context document, not duplicated in this model card.

Files

• chessformer-tiny-step30000.pt — the checkpoint (55,281,955 bytes)
• app.py, requirements.txt, chessformer/ — a self-contained local Gradio UI to play against this checkpoint (see "Play it yourself, locally" above)
• retroactive_eval.py — the exact script that produced the eval reports below; also the reference implementation for loading this checkpoint
• model_report.json — architecture, resolved BitNet/MoE/Liquid, parameter accounting
• holdout_report.json, puzzle_report.json, elo_report.json — the three retroactive eval passes
• dataset_provenance.json — source, split policy, parse stats
• environment_snapshot.json — training hardware (RTX 5070) + eval hardware note
• architecture_parity_summary.json — drift/bug fixes vs. an earlier onefile implementation, self-verified via torch.allclose
• training_curve.png — loss / accuracy / grad-norm over the run

License

Apache License 2.0, same as the parent GitHub repo — see LICENSE. This applies to everything here: the code (app.py, chessformer/, retroactive_eval.py) and the checkpoint weights (chessformer-tiny-step30000.pt). The parent repo's model weights are otherwise proprietary (MODEL_LICENSE.md) — chessformer is an explicit, scoped exception to that policy for this independent side-project only; see MODEL_LICENSE.md's exception clause and the matching DECISIONS.md entry (2026-08-06) for the full reasoning.