nanochat-tokenizer-2B

Nanochat Tokenizer

This is the tokenizer from Andrej Karpathy's educational project nanochat. This is the first step from the speedrun.sh script.

Training

For now, we need to download the first ~2B characters of pretraining dataset using the dataset script in nanochat.

export NANOCHAT_BASE_DIR=".cache/nanochat"
mkdir -p $NANOCHAT_BASE_DIR
python -m nanochat.dataset -n 8

Then, we can train the tokenizer with vocab size ~2B characters of data

python -m scripts.tok_train --max_chars=2000000000

And finally, evaluate:

python -m scripts.tok_eval

Tokenizer training

timestamp: 2025-10-14 10:29:05

• max_chars: 2,000,000,000
• doc_cap: 10,000
• vocab_size: 65,536
• train_time: 52.9085
• num_special_tokens: 9
• token_bytes_min: 1
• token_bytes_max: 32
• token_bytes_mean: 6.9197
• token_bytes_std: 2.8748

Tokenizer evaluation

timestamp: 2025-10-14 10:29:10

Comparison with GPT-2

Comparison with GPT-4