tokenizer-256k-v2

OpenEuroLLM Tokenizer v2 (256k)

SentencePiece BPE tokenizer for the OpenEuroLLM flagship models. v2 is a full retrain of the v1 256k tokenizer on a larger, cleaner corpus with the language list driven from the canonical training-data-catalogue/languages file (no hardcoded list this time — see v1's Georgian-gap incident).

Highlights vs SOTA — multi-domain eval

Evaluation is a held-out 5-domain suite (8,600 samples total), designed to test more than Wikipedia prose. Each column is mean tokens-per-whitespace-word (lower = better).

Eval composition: 7,200 FLORES (36 langs × 200 parallel sentences, held-out) · 500 Python (codeparrot) · 200 MATH+GSM8K · 200 OpenAssistant chat (ChatML-wrapped) · 500 FinePDFs (5 langs).

Summary of v2-256k vs the field:

• #1 overall (1.90) — beats every SOTA tokenizer on the multi-domain average.
• #1 on multilingual prose by a wide margin (FLORES 1.79 vs Gemma 2.13, GPT-OSS 2.07).
• #1 on chat (1.44, ChatML tokens working as intended).
• Competitive on math/PDF (within 0.3 of the leader).
• Loses on Python code (3.24 vs Llama 2.60). Llama 3's tiktoken-based BPE is more code-aggressive; even the v2 whitespace tokens don't fully close that gap.

v2 vs v1: per-language deltas on FLORES-200

FLORES-200 has parallel sentences across all languages (semantically equivalent translations), so fertility differences here are pure tokenizer effect (no content drift). Same 256k and 128k models, bold = v2 better, English first then alphabetical.

v2-256k improves on 30/36 languages. Biggest wins: Georgian −20.10 (v1 was full byte-fallback; v2 has real script subwords), Latvian −1.07, Albanian −0.67. The few regressions (bg/cs/de/it/mk/pl/sk) are small (+0.05 to +0.24) and reflect that v2 spread vocabulary across more code/whitespace coverage. Note: lb/ru/cy aren't tested here — they're not in the catalogue and were dropped from v2's training scope.

Language coverage

36 catalogue languages: bg, bs, ca, cs, da, de, el, en, es, et, eu, fi, fr, ga, gl, hr, hu, is, it, ka (new in v2), lt, lv, mk, mt, nl, no, pl, pt, ro, sk, sl, sq, sr, sv, tr, uk.

Languages removed from v2 vs v1: lb (Luxembourgish), ru (Russian), cy (Welsh) — not in the OpenEuroLLM catalogue.

Training details

• Algorithm: SentencePiece BPE
• Vocab size: 262,144 (2^18)
• Normalization: identity (lossless)
• Byte fallback: enabled
• Corpus: 500 GB streamed/sampled from the OpenEuroLLM "baby" cycle release shards on LUMI (10 May 2026 packer), spanning dclm, nemotron-cc, finepdfs, finepdfs-edu, olmo-mix (wiki/arxiv/pes2o), starcoder, finemath-4plus, megamath (text-code-block, web-pro), hplt-3.0, nemotron-cc-opus-1.1, nemotron-cc-tower+-0.1.
• Mix: ~70% English / ~7% code / ~5% math / ~18% other-langs (35 catalogue non-English languages, equal allocation).
• Character coverage: 0.9995
• Max piece length: 16

Special tokens

Core (locked at fixed IDs, in-vocab):

This fixes a v1 bug where <pad> was tacked on at vocab_size+0 (262144), out-of-vocab for downstream code that asserts pad_token_id < vocab_size.

User-defined symbols (204 total) — new for v2

Whitespace family — better code efficiency (45 tokens)

Multi-character whitespace runs are reserved as single tokens so code with deep indentation doesn't burn a token per space. None of these were in v1; pretrained tokenizers without them tokenize a 32-space indent as 16+ tokens, v2 does it in 1.

Example:

>>> from transformers import AutoTokenizer
>>> tok = AutoTokenizer.from_pretrained("openeurollm/tokenizer-256k-v2")
>>> python_code = "def f():\n        return 1"  # 8-space indent
>>> len(tok.encode("def f():\n        return 1", add_special_tokens=False))
# v2: tokenizes the 8-space indent as ONE token
# Llama 3.1: tokenizes it as ~8 separate tokens (one per space-pair)

StarCoder-style code corpus markers (16 tokens)

Reserved for code-corpus-formatted inputs (<filename>foo.py\n... <file_sep>\n<reponame>OpenEuroLLM/x\n...) — biggest single contributor to StarCoder2's code quality:

<filename>, <reponame>, <file_sep>, <gh_stars>, <empty_output>, <issue_start>, <issue_comment>, <issue_closed>, <jupyter_start>, <jupyter_text>, <jupyter_code>, <jupyter_output>, <jupyter_script>, <commit_before>, <commit_msg>, <commit_after>.

v1 had none of these.

Fill-in-the-middle (FIM) — code completion (4 tokens)

Full 4-token FIM set for code-completion training. v1 had 3 (missing <fim_pad>).

<fim_prefix>, <fim_middle>, <fim_suffix>, <fim_pad>

Chat formats (4 tokens)

ChatML as the modern default; Gemma-style retained from v1 for compatibility.

Tool use (2 tokens)

<tool_call>, </tool_call> (carried over from v1).

Reasoning / chain-of-thought (2 tokens) — new for v2

<think>, </think> — DeepSeek-R1 / Qwen3 convention. Reserved up-front so future thinking-style post-training doesn't need to retokenize.

Multimodal (3 tokens)

<start_of_image>, <end_of_image>, <image_soft_token> (carried over from v1).

Reserved for future (128 tokens)

<unused_0> … <unused_127> — forward-compat slots. v1 had 100; v2 expands to 128 (Llama 3 reserves 256, this is a middle ground at ~0.05% of vocab).

v1 vs v2 special-tokens summary

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("openeurollm/tokenizer-256k-v2")   # -> LlamaTokenizerFast

ids = tok("Hello world! Bonjour le monde.")["input_ids"]    # <bos> prepended
text = tok.decode(ids, skip_special_tokens=True)

The tokenizer is configured with add_bos_token=True and add_eos_token=False, so encoding prepends <bos> (id 1) and does not append <eos> (id 2).

Megatron-LM

--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model openeurollm/tokenizer-256k-v2 \
--append-eod

--append-eod appends <eos> as the document separator. Use --tokenizer-type HuggingFaceTokenizer, not Llama2Tokenizer.

Or with SentencePiece directly:

import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("tokenizer.model")
ids = sp.EncodeAsIds("გამარჯობა მსოფლიო")  # "Hello world" in Georgian

Both paths produce identical ids, verified over 18,575 samples spanning 36 languages, code, math, chat and PDF text. The one intended difference is that transformers extracts the core specials from raw text, so a document containing the literal string <eos> encodes to id 2; SentencePiece encodes it as text. Pass split_special_tokens=True to opt out.

Files

• tokenizer.json — HF fast tokenizer; this is the file transformers loads
• tokenizer.model — SentencePiece BPE model
• tokenizer.vocab — vocabulary listing
• special_tokens_map.json — HF special tokens map
• tokenizer_config.json — HF tokenizer config

Citation

Built for the OpenEuroLLM project (Horizon Europe). Source repo: https://github.com/OpenEuroLLM/tokenizer.