tokenizer-128k-v2

OpenEuroLLM Tokenizer v2 (128k)

128k-vocab variant of openeurollm/tokenizer-256k-v2. Same training corpus, same special tokens, smaller vocab for compute-constrained deployments.

Highlights vs SOTA — multi-domain eval (128k class)

Evaluation is a held-out 5-domain suite (8,600 samples total). Each column is mean tokens-per-whitespace-word (lower = better).

For the full SOTA comparison incl. 256k-class tokenizers, see tokenizer-256k-v2.

v2-128k vs the field:

• #1 overall (2.09) in its class — beats Mistral Nemo, EuroLLM, DeepSeek, Llama 3.2.
• #1 on multilingual prose (FLORES 2.00 vs Mistral 2.20, EuroLLM 2.21).
• #1 on chat (1.52, ChatML tokens working).
• Loses on Python code (3.32 vs Llama 2.60); Llama's tiktoken-based BPE is more code-aggressive.

v2 vs v1: per-language deltas on FLORES-200

FLORES-200 has parallel sentences across all languages (semantically equivalent translations), so fertility differences here are pure tokenizer effect (no content drift).

v2-128k improves on 29/36 languages. Biggest wins: Georgian −19.63, Latvian −0.98, Albanian −0.68. The few regressions (bg/cs/de/it/mk/pl/sk/uk) are small (+0.02 to +0.31). Note: lb/ru/cy aren't tested — they're not in the catalogue and were dropped from v2's training scope.

Language coverage

Same as 256k-v2: 36 catalogue languages, Georgian (ka) included, lb/ru/cy dropped.

Training details

• Algorithm: SentencePiece BPE
• Vocab size: 131,072 (2^17)
• Normalization: identity (lossless), byte fallback enabled
• Corpus: identical to 256k-v2 (500 GB, catalogue-driven multi-lingual mix)

Special tokens

Core (locked at fixed IDs, in-vocab):

Fixes v1's pad-out-of-vocab bug (v1 had <pad> at vocab_size+0 = 131072).

204 user-defined symbols reserved — same set as 256k-v2:

Why the whitespace tokens matter for code

A Python file with an 8-space indent tokenizes as 1 token in v2 (it's reserved as " ") but as 4-8 tokens in v1 and most other tokenizers. For a code-heavy training run this is a meaningful efficiency gain — see the 256k-v2 card for the example + full breakdown.

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("openeurollm/tokenizer-128k-v2")   # -> LlamaTokenizerFast

ids = tok("Hello world! Bonjour le monde.")["input_ids"]    # <bos> prepended
text = tok.decode(ids, skip_special_tokens=True)

The tokenizer is configured with add_bos_token=True and add_eos_token=False, so encoding prepends <bos> (id 1) and does not append <eos> (id 2).

Megatron-LM

--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model openeurollm/tokenizer-128k-v2 \
--append-eod

--append-eod appends <eos> as the document separator. Use --tokenizer-type HuggingFaceTokenizer, not Llama2Tokenizer.

Or with SentencePiece directly:

import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("tokenizer.model")
ids = sp.EncodeAsIds("გამარჯობა მსოფლიო")  # "Hello world" in Georgian

Both paths produce identical ids, verified over 18,575 samples spanning 36 languages, code, math, chat and PDF text. The one intended difference is that transformers extracts the core specials from raw text, so a document containing the literal string <eos> encodes to id 2; SentencePiece encodes it as text. Pass split_special_tokens=True to opt out.

Files

• tokenizer.json — HF fast tokenizer; this is the file transformers loads
• tokenizer.model — SentencePiece BPE model
• tokenizer.vocab — vocabulary listing
• special_tokens_map.json — HF special tokens map
• tokenizer_config.json — HF tokenizer config

Citation

Built for the OpenEuroLLM project (Horizon Europe). Source repo: https://github.com/OpenEuroLLM/tokenizer.