chess-transformer-437m-ft3h

Chess Transformer 437M — FT3h (best Elo)

Greedy policy checkpoint from the exp191 meta-attention line. Best measured Elo in this project: ~1950 (Stockfish LimitStrength gauntlet).

Play checkpoint: use greedy argmax over legal moves (temperature 0). Weak MCTS settings previously hurt Elo vs greedy.

Architecture

• 437M params (hidden_dim=1280, num_layers=18, num_heads=20)
• Strengthened board encoder + meta-factored attention + Shaw-on-pos
• Compact move vocab (1968), STM-normalized fused board tokens
• Policy + 3-class value head
• Trained with NorMuon + PolarExpress, torch.compile, cautious WD

Elo (greedy, movetime 0.05s SF side)

Estimate ~1950 (bracketed by SF1900 / SF2050). Black is weaker at high SF; French / English / Réti openings are relative weak spots. Losses are mostly tactical checkmates.

Training lineage

exp191 pretrain / early soft FTs
  → exp191_soft_ft3_lichess
  → exp191_soft_ft3b_unseen          (~1832 Elo)
  → exp191_soft_ft3h_edge_end  ★     (~1950 Elo)   ← this upload

Later soft / loss / tactics corrective FTs (ft3e/f/g/i, loss_fix, blunder_hard, tactics_hard) regressed Elo and were discarded. Do not prefer them over this file.

FT3h recipe (this checkpoint)

Data mixture used at FT3h

1. Shallow soft MultiPV (~2.0M train) — outputs/exp186_sf_multipv/soft_cache.pt
Stockfish MultiPV=8 soft targets on general positions (exp186 harvest from Lichess/SF-labeled position corpora such as avewright/chess-positions-lichess-sf / avewright/chess-positions-sf-labeled). Also mirrored on HF as dataset work under avewright/chess-soft-multipv-lichess.

2. Deep soft mix (~3.24M train) — outputs/unseen_quality_mix_ft3h/soft_cache.pt
Built to avoid overlap with prior FT caches (~33.9M excluded keys):

3. Hard ballast (~1.0M / step mix) — hard_ballast_d15_n2000000_s42.pt
Hard CE on Stockfish best-move labels, min depth ≥15, for stability when soft frac < 1.

Prior stage data (inherited via FT3b init)

FT3b was fine-tuned from ft3_lichess on:

• Same shallow exp186 soft cache
• Deep: soft_cache_unseen_ft3.pt (~3.2M Lichess-eval soft, held out from earlier FT3)
• Soft α 0.55, deep mix 0.55, hard ballast

Weights therefore already encode earlier Lichess MultiPV soft pretraining before the FT3h edge/end-lean mix.

Related public datasets

• avewright/chess-soft-multipv-lichess — soft MultiPV caches
• avewright/chess-soft-syzygy — syzygy soft (~498k)
• Lichess/chess-puzzles — puzzles (used in later experiments; not a major FT3h spice contributor)

Load

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("avewright/chess-transformer-437m-ft3h", "best_model.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
state = {k.replace("_orig_mod.", ""): v for k, v in ckpt["model_state_dict"].items()}
config = ckpt["config"]  # dict → ChessTransformerConfig(**config)
# build_model(config); load_state_dict(state); vocab_version = ckpt.get("vocab_version", "compact")

Requires this repo’s ChessTransformer / compact move vocab (MOVE_VOCAB_VERSION=compact).

License

Weights: MIT (same as project). Upstream Lichess eval / puzzle data retain their respective terms (puzzles: CC0).