picur-tokenizer

picur-tokenizer

A 49,152-piece SentencePiece Unigram tokenizer for Hungarian, fitted so that it segments on real morpheme boundaries, rather than on whatever the compression objective prefers.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("picur/picur-tokenizer")
tok.tokenize("a házakban lakó emberekkel")
# ['▁a', '▁ház', 'ak', 'ban', '▁lakó', '▁ember', 'ek', 'kel']

Why another tokenizer

• Nevezz meg három állatot.
• Kutyácska, lovacska, verebecske.
• Jó, de kicsinyítő képző nélkül.
• Kecs, macs, szar.

Hungarian is agglutinative: one stem carries a long tail of suffixes, so ház (house) surfaces as házban, házakban, házaimban. A tokenizer fitted on compression alone shatters that paradigm across unrelated rows, and the model then has to learn ház several times over - once per spelling of the same word. Training effort that should have gone into knowledge goes into re-learning morphology instead.

This vocabulary is fitted to keep a stem on one row and let the suffixes be their own pieces, so a paradigm shares its knowledge instead of spreading it.

How it segments

Vowel harmony is why ban/ben, kel/kal and hoz/hez/höz all exist as separate rows: they are the same grammatical function wearing different vowels. See Tied rows for what is shipped to help a model learn them as one.

Measured

Every number below is reproducible from public artifacts by benchmark.py in this repository:

pip install transformers datasets
python benchmark.py

It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five published tokenizers. Nothing from the pipeline that produced this vocabulary is involved.

Paradigm sharing

The property this vocabulary is built for. On UniMorph Hungarian first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs:

one stem row + suffixes asks both halves of the question at once: the lemma is a SINGLE row, and its inflected form begins with that row. So every mention of the word teaches one embedding.

Neither half can be won on its own. Cutting everything small satisfies prefix kept and destroys lemma is one token. Swallowing whole inflected forms does the reverse. Only the conjunction says a paradigm shares a row, and it does so at the same token cost per form as the closest baseline.

Fertility

Characters a token buys, on the first 2,000 documents of each dataset:

On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a defect. PULI buys more characters per token by cutting házakban into ház akban, keeping stem and endings fused; this vocabulary spends one more token to cut it into ▁ház ak ban, and gets the paradigm table above in return.

Where the comparison is not about Hungarian morphology it inverts: on mathematics and code this vocabulary beats both Hungarian baselines, because a slice of each was in the fit corpus so technical text does not fall back to bytes. Qwen wins those outright, as an English-first tokenizer should.

Special tokens

Tool-call markers (<tools>, <tool_call>, <tool_response> and their closers) are present as ordinary pieces, following the Hermes convention.

Note the ChatML split: <|im_end|> terminates a turn, <|eos|> terminates a sequence. A chat model should stop on <|im_end|>.

Tied rows

ties.json maps 1,357 allomorph rows onto the one row a model should learn first: ban -> ben, höz -> hoz, across 100 suffix families. Vowel harmony makes these variants of one morpheme, so initialising their embeddings from a shared base gives the model the paradigm for free rather than making it discover the relationship from data.

This is an initialisation aid, not part of tokenization, nothing reads it automatically.

How it was fitted

The vocabulary is overfitted well past its target size and then selected down, with a declared Hungarian lexicon deciding what survives:

1. Seed - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities, 1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
2. Overfit - a large Unigram vocabulary is trained on the corpus.
3. Evict - any piece that is a regular inflection of a declared headword is removed. magyarokat becomes ▁magyar ok at: the whole ending okat is not a row, while the plural ok and the accusative at each are, and every other paradigm reuses them.
4. Spare - an eviction is reversed if the word would then segment worse than its morphology says it should.
5. Prune - rows the corpus never reaches are dropped.

Corpus: 700 Mchar drawn across 57 Hungarian sources: encyclopedic, news, legal, parliamentary, academic, literary, subtitles, web and forum. Plus small code and mathematics slices, so that technical text does not fall back to bytes.

Limitations

• Hungarian-first. English and code are handled but not optimized; other languages fall back to bytes.
• Coverage, not rules, sets the floor. A stem outside the 28,000-stem seed can still be mis-split, and an alternating stem can lose on score: művek cuts as mű + vek where morphology says műv + ek.
• Compound seams come from a declared lexicon, so a compound absent from it gets no guaranteed seam.
• Cased entity handling is partial. 1,000 capitalised forms are seeded; beyond those, capitalisation costs a row.
• Two losses are declared, not accidental. Metaspace writes the word mark as U+2581, so a literal U+2581 in the input is indistinguishable from a space. The tokenizer declares a Replace(U+2581, " ") normalizer ahead of Metaspace, which makes the collision part of the contract instead of a surprise - text carrying block-drawing characters (a sparkline, say) round-trips with that one character turned into a space. Leading whitespace on an input is dropped for the same reason: Metaspace prepends a mark unconditionally. Everything else round-trips byte-exactly, including every script outside Latin, via byte fallback.

Attribution

This tokenizer is a derivative work of the following resouces, see NOTICE.

• morphdb.hu - CC-BY 2.5, (c) Nemeth Laszlo and Godo Ferenc. The declared lexicon: 351,917 headwords, 37,042 compound seams, alternation classes and vowel harmony. This decided what the fit protects and what it evicts.
• hunmorph-foma - BSD-2-Clause, (c) 2016 r0ller. Finite-state transducers that analysed corpus words into lemma plus tags, bootstrapping the segmentation this vocabulary is scored against.
• UniMorph Hungarian - CC BY-SA 3.0, derived from English Wiktionary. Used by benchmark.py only: it is scored against and never built from, which is what makes the paradigm table above independent of everything that produced the vocabulary.

The tokenizer itself is Apache-2.0.

Citation

@misc{picur-tokenizer,
  title  = {picur-tokenizer: a morphology-aware Hungarian tokenizer},
  author = {Bencz, Lajos},
  year   = {2026},
  url    = {https://huggingface.co/picur/picur-tokenizer}
}