boundary-markers-ko-d12-plain-bpe

Boundary markers, Korean, 12 layers: plain tokenizer, BPE training

Three language models (seeds 0, 1 and 2) trained to compare subword vocabularies that mark word boundaries explicitly, the schemes of Explicit Boundary Markers for Subword Vocabularies (Sander Land and Clara Meister). The paper reports English; these are Korean models trained the same way in September 2026. Models within a language differ only in their tokenizer.

Tokenizer

SCRIPT-encoding pretokenization with no boundary markers. This is the baseline.

Trained with BPE on a 5 GB sample of Korean FineWeb, with a total vocabulary of 34,685. The models' vocabulary is 34,686: these entries plus a beginning-of-sequence token. File: tokenizer/fineweb_ko_5gb_quick_plain_bpe_v34685.json.gz, sha256 3dbf1e06e409e362be5bc29fd059df3470df124512e7a41e2d3011006893a59c. To load it, check out script_tok and run from its root:

from huggingface_hub import hf_hub_download
from paper_utils.boundary.downstream.boundary_tokenizer import BoundaryBPETokenizer as Tok
path = hf_hub_download("cmeister/boundary-markers-ko-d12-plain-bpe", "tokenizer/fineweb_ko_5gb_quick_plain_bpe_v34685.json.gz")
tok = Tok.load(path)

Training

nanochat at commit 92d63d4, launched with paper_utils/boundary/downstream/run_arms.sh from script_tok: 12 layers, width 768, 6 attention heads, context 2,048 tokens, 2,553 steps of 524,288 tokens (1.34 billion tokens), one GPU per model.

The training text is 3 shards of Korean FineWeb-2, from the fineweb-2_0_1-quality_10-filterrobots release, 1.22 billion characters, read about 3.1 times. The seed sets the weight initialization and the order of the shards, and the order for a given seed is the same for every tokenizer, so models with the same seed are directly comparable.

With only 3 shards, seeds 1 and 2 drew the same shard order. Those two models differ in weight initialization alone, so the three seeds cover two data orders, not three.

Results

Validation bits per byte: the summed loss on a held-out Korean FineWeb-2 shard divided by the true UTF-8 length of the scored text. Lower is better. Values are comparable within a language only.

This is the baseline the other schemes are compared with.

Three seeds give a direction rather than a precise estimate. The full comparison across schemes, trainers and languages is in the script_tok repository.

Files

• seed<n>/model_002553.pt: the final weights, a PyTorch state dict that loads with torch.load(..., weights_only=True).
• seed<n>/meta_002553.json: nanochat's model and training configuration.
• seed<n>/train.log: the full training and evaluation log.
• seed<n>/archive.json: sha256 of each file as copied from the training run.
• tokenizer/: the tokenizer, identical for all three seeds.