pokemon_gpt1

ptcg-gpt

A 30M-parameter GPT trained on Pokémon TCG game-event streams from the Kaggle pokemon-tcg-ai-battle ladder replays. The backbone is the unmodified transformer block stack from Andrej Karpathy's nanoGPT (see NOTICE.md); the input embedder and output heads are replaced to consume/produce structured game-stream tokens instead of text. Two checkpoints ship:

The training data is per-seat, append-only token streams serialized from real ladder episodes (stream format v2, 18 float features per token — path9_gpt/STREAM_SPEC.md is the normative contract). One document = one (game, seat), averaging ~2.4–2.7k tokens per seat-game across the shipped corpora (individual seat-games range from ~150 to ~6k).

What this repo can and cannot do. It can ingest replays, train, finetune, and evaluate on held-out data. It cannot play games: the competition game engine (cg.dll), the self-play driver, the Kaggle submission exporter, and the arena all deliberately stay in the parent research repo. path9_gpt/agent.py ships as the reference inference implementation (incremental serializer + KV-cache), but there is nothing here to plug it into.

Results

Training (both on one RTX 3060):

Playing strength, measured in the parent repo's arena (not reproducible here — no engine):

Caveats that apply to any number you produce yourself: 100-game win rates carry roughly ±10 pp of noise — use 300+ games before claiming anything; and argmax-vs-argmax mirror matches between behavior-cloned siblings are a degenerate evaluation (they swing wildly and mean nothing).

Quickstart

pip install -r requirements.txt
python -m pytest                      # from the repo root; needs torch

Load a checkpoint:

from path9_gpt.model import load_model

model = load_model("models/sft_30m_v1/best.pt").eval()
# checkpoints are self-describing {"state_dict", "config"} dicts (fp32, model-only)

Smoke-train on the bundled sample corpus (108 episodes, 522k tokens, mid tier). Commands are kept on one line so they paste into PowerShell as well as bash:

python -m path9_gpt.pack --data data/sample     # regenerate data/sample/packed/
python -m path9_gpt.train_pretrain --data data/sample --out runs/smoke --preset 7M --epochs 1 --lr 3e-4 --min-lr 3e-5 --batch-tokens 16384 --micro-batch-tokens 8192 --ctx-dropout 0.35 --device auto

(Add --n-layer 2 --n-embd 64 --n-head 2 to override the preset and make it near-instant on CPU.)

Finetune from a shipped checkpoint on your own data:

python -m path9_gpt.train_sft --init-from models/sft_30m_v1 --data <your_top_tier_dir> --out runs/sft_v2 --epochs 2 --lr 1e-4 --min-lr 1e-5 --warmup-frac 0.03 --batch-tokens 65536 --micro-batch-tokens 8192 --ctx-dropout 0.35 --val-episodes models/val_episodes_v1.txt --device cuda

Always pass --val-episodes models/val_episodes_v1.txt — it pins the shared holdout the shipped numbers were measured on, so those episodes never leak into your training set and your val numbers stay comparable. See docs/FINETUNING.md for the full ingest → pack → train walkthrough and docs/DESIGN.md for the architecture and the lessons/pitfalls list.

Repo layout

Note for GitHub

Each best.pt is 121 MB fp32, which is over GitHub's 100 MB per-file hard limit. This repo is therefore initialized with git-lfs tracking models/**/*.pt (see .gitattributes), so pushing to GitHub works as-is — but anyone cloning needs git-lfs installed to get real checkpoints instead of LFS pointer files. If you would rather not depend on LFS, run scripts/shrink_fp16.py to produce an fp16 copy (~61 MB, fits under the limit; fp16 rounding changes logits marginally — keep the fp32 original around for training resumes and exact reproduction) and commit that instead.

Licensing

• This repo has no license yet. Default copyright applies (all rights reserved) — pick a license before publishing anywhere public.
• The transformer blocks inside path9_gpt/model.py derive from nanoGPT and are MIT-licensed — attribution and license text in NOTICE.md.
• The training data derives from Kaggle ladder replays of the pokemon-tcg-ai-battle competition (this includes data/sample/ and data/raw_episodes_sample/). Check the competition's data/terms of use before redistributing any of it or models trained on it beyond this circle.