boundary-markers-ko-d12-bnd_wpd_caps-mingram

Boundary markers, Korean, 12 layers: bnd_wpd_caps tokenizer, MinGram training

Three language models (seeds 0, 1 and 2) trained to compare subword vocabularies that mark word boundaries explicitly, the schemes of Explicit Boundary Markers for Subword Vocabularies (Sander Land and Clara Meister). The paper reports English; these are Korean models trained the same way in September 2026. Models within a language differ only in their tokenizer.

Tokenizer

As bnd_wpd, plus case codes: a title-case word is written as <^> followed by its lowercase form, and an all-caps word as <^^> followed by its lowercase form, with the code outside the word's markers. The case test requires a cased character, so in a script without case the codes do not fire.

Trained with MinGram on a 5 GB sample of Korean FineWeb, with a total vocabulary of 34,685. The models' vocabulary is 34,686: these entries plus a beginning-of-sequence token. File: tokenizer/fineweb_ko_5gb_quick_bnd_wpd_caps_mingram_v34685.json.gz, sha256 7dcddc1f1e661a2393ef9ec17cebca0d066320f27d55e3265f7d22f335aeb8f4. To load it, check out script_tok and run from its root:

from huggingface_hub import hf_hub_download
from paper_utils.boundary.downstream.boundary_tokenizer import BoundaryMinGramModel as Tok
path = hf_hub_download("cmeister/boundary-markers-ko-d12-bnd_wpd_caps-mingram", "tokenizer/fineweb_ko_5gb_quick_bnd_wpd_caps_mingram_v34685.json.gz")
tok = Tok.load(path)

Training

nanochat at commit 92d63d4, launched with paper_utils/boundary/downstream/run_arms.sh from script_tok: 12 layers, width 768, 6 attention heads, context 2,048 tokens, 2,553 steps of 524,288 tokens (1.34 billion tokens), one GPU per model.

The training text is 3 shards of Korean FineWeb-2, from the fineweb-2_0_1-quality_10-filterrobots release, 1.22 billion characters, read about 3.1 times. The seed sets the weight initialization and the order of the shards, and the order for a given seed is the same for every tokenizer, so models with the same seed are directly comparable.

With only 3 shards, seeds 1 and 2 drew the same shard order. Those two models differ in weight initialization alone, so the three seeds cover two data orders, not three.

Results

Validation bits per byte: the summed loss on a held-out Korean FineWeb-2 shard divided by the true UTF-8 length of the scored text. Lower is better. Values are comparable within a language only.

Mean difference -0.00208, standard deviation across seeds 0.00131. A positive difference means this scheme scored lower than plain.

Three seeds give a direction rather than a precise estimate. The full comparison across schemes, trainers and languages is in the script_tok repository.

Files

• seed<n>/model_002553.pt: the final weights, a PyTorch state dict that loads with torch.load(..., weights_only=True).
• seed<n>/meta_002553.json: nanochat's model and training configuration.
• seed<n>/train.log: the full training and evaluation log.
• seed<n>/archive.json: sha256 of each file as copied from the training run.
• tokenizer/: the tokenizer, identical for all three seeds.