tokenizer-lm-ablations

Tokenizer ablations: 231 language models trained from scratch

231 language models that share an architecture, a training corpus, a document order and a hyperparameter set, and differ in the tokenizer they were trained with. 98 tokenizers are involved, of which 94 are released here and 4 are off-the-shelf tokenizers cited rather than redistributed. Every model was trained for this study, not adapted from an existing checkpoint.

Acknowledgments: These analyses were part of the development for the Apertus 2.0 tokenizer, funded by the Swiss AI Initiative.

The models back three written works. A model can appear in more than one.

Training regimes

Loading a model

Each model lives in its own subfolder. The architecture is not one of the built-in transformers architectures, so loading needs trust_remote_code=True; the model class ships in each subfolder as nanochat_model.py.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "cmeister/tokenizer-lm-ablations", subfolder="models/full-128k-allmulti-equal-bpe",
    trust_remote_code=True, dtype="float32")
tokenizer = AutoTokenizer.from_pretrained("cmeister/tokenizer-lm-ablations", subfolder="models/full-128k-allmulti-equal-bpe")

text = "def add(a, b):\n    return a + b\n"
ids = [0] + tokenizer(text)["input_ids"]   # training prepends BOS

Training prepended the beginning-of-sequence token to every document, and the released tokenizer files carry no post-processor that would do this on their own. Prepend the id yourself, as above, or the model receives input it never saw in training. Each model card states its own BOS id.

What varies, and what does not

Within a training regime, every model sees the same documents in the same order with the same optimizer settings. The tokenizer changes the token sequence those documents produce, so token counts, sequence packing and the effective amount of text per context window differ between models. That is the effect under study, not a confound to remove.

Ablation grid: the tokenizer axis

One row per tokenizer. pretokenizer names the regex constant in the training script; normalizer and vocab are read from the released tokenizer file itself. In the names, gpt4o, claude and gpt2 refer to a borrowed pretokenization regex, not to those vendors' tokenizers.

n/a in the pretokenizer, normalizer or training-data columns means the tokenizer was not built by this repository's training script and has no row there. Its build settings are in its own card, taken from the build manifest the external trainer wrote or from the model registry note.

Ablation grid: the model axis

The per-model settings and headline metrics are in ablation_grid.csv (231 rows) and models.json. Each model's own card repeats its row. The columns are the architecture (layers, width, heads, vocabulary, value-embedding dimension, soft cap), the training point (step, token budget, seed) and the evaluation results.

Architecture

nanochat (Karpathy), MIT licensed, with three patches applied for this study (branch tokenizer-lm-patches, commit f919449):

1. ve_dim: the per-layer value-embedding tables are built at a reduced dimension and projected up to the key and value dimension by a learned matrix, instead of being built at full width. All released models use ve_dim=128. Without this, a 128k vocabulary would spend most of the parameter budget on those tables.
2. logit_softcap: the logit soft cap is a config field where 0 disables it, instead of a hardcoded 15. Released models use 15 unless their card says otherwise.
3. width_lr_exponent: the AdamW learning-rate width scaling exponent is configurable, set to -1.0 for muP. This affects training only.

The rest is nanochat as published: rotary embeddings, QK norm, ReLU squared MLP, sliding-window attention on three of every four layers, a smear gate that mixes the previous token, a backout term at the middle layer, per-layer residual and x0 scalars, parameterless RMSNorm, and untied input and output embeddings.

Each released checkpoint was checked against the training-time model: the same random token ids run through both give the same logits in float32, and the top-1 prediction agrees at every position.

Training data

One mixture, used for both tokenizer training and language-model training: 35% English web text (FineWeb-Edu), 30% multilingual text over 30 languages and 8 scripts (a quality-filtered FineWeb-2), 15% mathematics (FineMath 4plus), 15% code (StarCoderData, highest quality tier). Multilingual weights are proportional to each language's estimated character count in the source, so low-resource languages get less data, which is the resource gradient the cross-lingual study measures.

The math and code regime continues from the same corpus family with a mixture weighted toward mathematics and code. Code evaluation uses a StarCoderData tier held out from training.

Upstream licenses, as their dataset cards state them: FineWeb-Edu ODC-By, FineWeb-2 ODC-By, FineMath ODC-By, StarCoderData listed as other.

What is not here

Models whose tokenizer was later found defective are not released. Two defects were found during the study: a tokenizer family that stripped newline and carriage-return characters and had an incomplete byte alphabet, and a SuperBPE stage-2 pretokenizer that split combining marks its stage 1 had kept together, which breaks Bengali, Devanagari, Tamil, Thai and Arabic sequences. The affected checkpoints stay on the training cluster with their defect recorded. Where a slug names both a defective original and a later fixed retrain, the released model is the retrain, identified by comparing the tokenizer file against the archived defective copy.

A separate family of runs trained on a quota-based corpus was retired for a cross-lingual confound rather than a defect. Those are not released either.

Files

Citation

TokEval: A Tokenizer Evaluation Suite. Clara Meister. COLM 2026.