letterpress

letterpress - trained character-level GPTs

The trained checkpoints of letterpress, a character-level GPT built from scratch in PyTorch (following Karpathy's "Let's build GPT" lecture) and trained on nested tiers of a public-domain book corpus. These are base models: they continue text in the style of the books they read, one character at a time - they are not chatbots and do not follow instructions.

(The checkpoint files keep the model family's historical name byob-lm - the letterpress repo was formerly byob_llm - so paths, benchmark reports, and the companion dataset stay stable.)

Models

Held-out bpc = bits-per-char on public-domain books absent from every training tier; wikitext = out-of-domain. Both measured by the repo's bundled lm_bench harness (seed 1337, byte-normalized); full scorecards live in the repo under lm_bench/benchmarks/. Lower is better; the corpus tiers are nested, so every gain down the table is scale, not data luck.

How to use

The checkpoints are self-describing ({model, config, stoi, itos}, optimizer state stripped) but unpickle the letterpress GPTConfig, so you need the repo's source next to them:

git clone https://github.com/Novotarskyi/letterpress.git && cd letterpress
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
hf download disco-jack-basement/letterpress 2xlarge/byob-lm.best.pt --local-dir models

.venv/bin/python -m inference.sample   --ckpt models/2xlarge/byob-lm.best.pt --prompt "ROMEO:"
.venv/bin/python -m inference.interact --ckpt models/2xlarge/byob-lm.best.pt
cd lm_bench && ../.venv/bin/python -m lm_bench run --model byob:../models/2xlarge/byob-lm.best.pt --tasks core

Training provenance

• nano - Tiny Shakespeare (1.1M chars); the lecture demo, trained locally
• medium - medium tier (523M chars); Apple-Silicon MPS, ~1 h
• large - large tier (1.08B chars); Apple-Silicon MPS, ~3.5 h
• xlarge - xlarge tier (2.05B chars); rented H100 SXM, ~2.5 h
• 2xlarge - 2xlarge tier (4.26B chars); rented H200, ~12.6 h

Each <tier>/ folder here also carries the archive's provenance files (context.md with the full hyperparameters and val curve, corpus_stats.txt, corpus.lock.json, corpus_index.md) where the archive has them.

Data

Trained exclusively on the companion public-domain corpus disco-jack-basement/byob-pd-book-corpus (CC0; Project Gutenberg, Standard Ebooks, Internet Archive, Wikisource), curated under a strict, code-enforced no-Russian-content rule.

License

Code and weights: MIT. The training corpus is CC0 and published separately.