wyrd42-tokenizer

WYRD42 — BPE-32k Tokenizer

A deterministic 32,768-token ByteLevel BPE tokenizer for WYRD42, a seeded, induction-resistant constructed language. It is the shared token id-space for both base-pretraining and chat fine-tuning of a 1.1B WYRD language model.

What is WYRD?

WYRD is a constructed language defined not by a hand-written grammar but by a seeded deterministic generator: a pure program plus a private 256-bit master seed that coins its own lexicon, fills every inflectional paradigm, and realises surface strings in linear forward time. Irregularity across orthography, morphophonology, morphology, and syntax is keyed to a pseudorandom function of (lexeme, morphosyntactic cell) under the seed, so the corpus is cheap to generate forward and engineered to be intractable to invert into a compact grammar. Seed for this corpus: "42" (hence WYRD42).

Determinism & reproducibility

The tokenizer is a byte-verifiable function of the corpus and a pinned trainer:

• Training sample: a fixed, sorted, per-register byte budget of the wyrd field of the seed-42 v4 corpus — no sampling, no shuffling (train/train_tokenizer.py).
• BPE trainer: tokenizers==0.22.2, vocab_size=32768, min_frequency=2, ByteLevel.
• manifest.json records the sha256 of the sample and of vocab.json / merges.txt, so a re-run can be byte-compared.

Special tokens

<|endoftext|> · <|doc|> · <|q|> · <|user|> · <|assistant|> · <|pad|> — document, query, and chat-role markers, so base bins and chat-SFT data share one id-space.

Fertility

3.38 characters / token (3.92 bytes/token) on native WYRD. WYRD is agglutinative with long words; this fertility sets the ~30B-token training budget for the 1.1B model.

Registers in the training sample

FineWeb-Edu, FineWeb (commerce), Wikipedia, Gutenberg, balanced native-generation, conversations, queries, and a multilingual feature-rich supplement — so the merges cover web, narrative, chat, and the full WYRD feature space (evidentiality, dual/paucal, etc.).

Usage

from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
ids = tok.encode("Riga-Kola thüükwúe John Muir").ids   # brands pass through as tokens
print(tok.decode(ids))                                 # -> "Riga-Kola thüükwúe John Muir"

Proper names and brands pass through the language verbatim as a transparent loanword channel, so a brand first seen after training still has a valid token sequence.

License

Research artefact. WYRD is a linguistics research object (forward-easy / invert-hard keyed text transform), not a security primitive.