SIL-v01

tinyshakespeare -- char tokenizer, n_embd=128

This is an educational project, not a practical language model. It is a from-scratch transformer built to learn the mechanics of language modeling end-to-end. It is not intended for any downstream or production use -- treat it as a worked example, not a tool.

Model architecture

Token + position embeddings -> n_layer stacked transformer blocks (pre-norm multi-head causal self-attention + feedforward, residual connections) -> final LayerNorm -> linear head to vocab logits. Full hyperparameters are in config.json; nothing about the architecture is hardcoded in the loader -- see load_model.py.

Training corpus

Trained on TinyShakespeare (~1.1M characters), fetched directly from https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt.

Results

Four runs, all on TinyShakespeare, same architecture family (n_layer=4, n_head=4, block_size=32, lr=1e-3, max_iters=8000, batch_size=32, seed=1337) — a controlled sweep over embedding width, plus one run swapping in a trained BPE subword tokenizer instead of character-level.

Raw cross-entropy (nats/token) is not comparable across tokenizers — a 1000-token BPE vocabulary and a 65-token char vocabulary have different random-guessing floors (ln(1000) ≈ 6.9 vs ln(65) ≈ 4.2), so a higher raw loss on BPE does not mean a worse model. Bits-per-character (BPC) normalizes both onto the same unit — bits of model surprise per character of the original text — and is the number to compare across the table.

Published checkpoint: char, n_embd=128 (bolded row) — lowest val BPC of the char-level sweep.

Scaling observation: doubling n_embd costs a roughly constant ~3.8× parameters each step (32→64, 64→128), but the val-BPC improvement shrinks each time (0.235 → 0.139 in BPC terms), and the train/val gap widens — diminishing, saturating returns from embedding width alone once depth (n_layer=4) and context (block_size=32) are held fixed.

Tokenizer observation: char-level (BPC 2.35) still edges out this one BPE run (BPC 2.42) at comparable parameter count, but BPE achieves that with ~7x fewer parameters than the 812K char model and visibly more coherent generated chunks — a fair head-to-head at matched parameter count wasn't run here.

Files

Usage

pip install torch safetensors
python load_model.py

load_model.py reads config.json for every architectural parameter -- it does not assume or hardcode them -- loads model.safetensors into a freshly constructed model, and samples 400 characters to prove the checkpoint and its config agree.

Limitations

This is a ~65-char-vocabulary, 32-token-context toy model trained for 8,000 steps. It reproduces surface Shakespeare-ish texture but not coherent meaning, plot, or factual content. Do not use this for anything beyond studying how these mechanics fit together.