small-llm-superbpe-8000-v2

Kefir SuperBPE tokenizer

The 8,000-ID tokenizer used for the Kefir v1 corpus, by Edoardo Avenia and Rocco Angelella. It was trained on 13 September 2026 from a filtered English seed. That seed used earlier prose filters; the later 100B corpus used a stricter filter.

Training had two stages: 7,192 subword entries, then 800 merges that may cross word boundaries. Eight special tokens complete the vocabulary. The end-of-document token is ID 7992. Pretokenization groups digits, splits them from the right into groups of three, then applies byte-level encoding without a whitespace regex so the second stage can learn cross-word tokens.

tokenizer.json is the artifact used to encode the corpus. Its SHA-256 is 068e20ef8a16b1bff2a2a1d161e5a6f1cfd0935befa7c0ae734c9ec376d373bb. The builder encodes text with IDs below 7992 and appends one end-of-document token after each document. Producer-side model files are also provided, but their segmentation is not identical on every input; use tokenizer.json for corpus parity.

The training record is in evaluation.json. Compression and round-trip behavior were measured; downstream model quality from this tokenizer alone was not isolated.

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("abcastor/small-llm-superbpe-8000-v2")