shakespeare-nanogpt-2

Model Card — shakespeare-nanogpt-2 (v2)

A sup computer release — a small language model studio. Model page · monorepo (frozen code: projects/shakespeare/models/shakespeare-nanogpt-2/, tag shakespeare-nanogpt-2) · run it in your terminal with the repo's sup CLI.

<div class="takeaways"> <p class="takeaways-label">Key takeaways</p> <ul> <li>The improved model: full corpus + modern architecture (RoPE, RMSNorm, bias-free) + GPT-2 BPE, reaching held-out <code>BPC 1.919</code> (−20% vs. the v1 baseline).</li> <li>The best of Experiment 01's four rounds (Round 3, early-stopped BPE). The Round 4 "champion" that stacked more regularization <strong>regressed</strong>.</li> <li>Still mimicry, and scores are single-seed — <strong>data is the ceiling</strong> the next version will have to raise.</li> </ul> </div>

The current best model in the shakespeare-nanogpt series: full corpus + modern architecture + BPE tokenizer, held-out BPC 2.395 → 1.919. It is the winner of a four-round LLM-assisted research experiment in which Claude Opus 4.8 acted as the researcher — diagnosing, changing, retraining, and measuring — while this small model was the thing being improved. This is the precursor stage to recursive self-improvement, not RSI itself. v2 is Round 3 of that experiment.

Series note. Successor to shakespeare-nanogpt-1. Both versions and the full story are in MODELS.md; the write-up is Experiment 01 (report index: research-docs/reports/) and the scoreboard is leaderboard.md.

Model details

Intended use

Same as v1 — a learning project, here additionally demonstrating LLM-assisted model development (a large model improving a small one, measured honestly). Generated text is higher-quality Shakespeare-styled mimicry than v1, but still not coherent or factual.

Out of scope: real use of the text; any presentation of output as genuine Shakespeare or as fact. No instruction following, no safety tuning.

Training data

The Complete Works of Shakespeare (~5 MB), ~5× the data of v1's Tiny Shakespeare. Prepared as GPT-2 BPE tokens (projects/shakespeare/data/shakespeare_full_bpe/). A fixed 250k-character held-out test set (projects/shakespeare/test.txt) that no model trains on is used for evaluation.

Training procedure

Trained with core/nanogpt_core/train.py on the same Apple Silicon MPS setup as v1 (~20 min). Note: the model overfits — validation loss bottomed around step ~1000 then rose; the save-best-val policy automatically kept the early, best checkpoint.

Evaluation

Scored on the fixed held-out test in bits-per-character (BPC) — a tokenizer-agnostic metric, so char-level and BPE models are directly comparable. Lower is better.

Three levers worked; the fourth backfired. The bars step down through Round 3, then tick back up at Round 4.

End to end: BPC 2.395 → 1.919, a 20% reduction. v2 is Round 3, which already combines all three productive levers.

Researcher efficiency

We also tracked Claude's own token cost per round, to ask how much intelligence each unit of improvement cost. Round 1 (fixing the data bottleneck) paid off hugely; later rounds returned far less for similar effort, and Round 4 spent the most and went backwards.

Charts generated by dataviz/.

Limitations

• Data-bottlenecked. Round 4 added dropout + longer training to push past R3 and regressed (1.919 → 1.947), samples collapsing into "Romeo, Romeo" loops. You cannot regularize away too little data — this is the most instructive result, and v2's ceiling is data.
• Overfits despite the larger corpus (see Training procedure).
• Single-seed measurements. Every score comes from one training run at a fixed seed (1337), with no variance estimate. The two smallest table deltas — the modern-architecture −1.6% and the champion's +1.5% regression — are within plausible run-to-run noise and should be read as directionally uncertain; the large gains (more data, BPE) are well outside any plausible noise. The seed is now a recorded --seed knob, and future versions report multiple seeds.
• Still mimicry: more fluent than v1, but no meaning, knowledge, or factuality; no instruction following or safety tuning.

How to use

# self-contained v2 folder (weights are gitignored — rebuild them)
cd models/shakespeare-nanogpt-2
python prepare.py     # downloads the Complete Works, BPE-encodes it here
python train.py       # -> ./ckpt.pt
python eval.py        # score on the shared held-out test (expect BPC ~1.919)
python sample.py --start="ROMEO:"

Citation / credits

• nanoGPT by Andrej Karpathy (MIT) — model + training code.
• The Complete Works of Shakespeare (public domain).
• LLM-assisted research experiment run with Claude Opus 4.8 (Claude Code) as researcher, human oversight.

---

Addendum — June 2026

Added in the site-standardization pass (ADR-0015). The card above was unchanged by that pass; this is a tracked addendum. (A later house-style pass, 2026-07-04, edited the card's prose — emphasis and sentence structure only, no facts.) Site-wide fixes — repo links now resolve to GitHub/site routes, code blocks render within the column — apply automatically.