Micka-65k

Micka-65k

A 65,536-token, lossless byte-level BPE tokenizer for Slovenian-focused models using English and Croatian. This repository contains a tokenizer, chat template, provenance and evaluation reports. It contains no language-model weights and cannot generate answers by itself.

• Author: Marko Kokol / klokedm.
• License: CC-BY-4.0. See LICENSE and ATTRIBUTION.md.
• Vocabulary: 65,536 IDs: 65,408 ordinary tokens and 128 special tokens.
• Training: 514,965,206 UTF-8 bytes, 198,223 records, 73.9412% source-labelled Slovenian.
• Independent review: 24,276/24,276 exact text round trips, zero unknown tokens, across 23 language/script configurations.
• Intended use: a compact tokenizer for a new or adapted Slovenian/English/Croatian model. Other European languages are representable, but often substantially less efficient than with Gemma's native tokenizer.

Quick start

pip install transformers jinja2

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("klokedm/Micka-65k", trust_remote_code=False)
text = "Slovenščina, English in hrvatski. Čšž, đ, € — 2026."
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, skip_special_tokens=False,
                        clean_up_tokenization_spaces=False) == text
assert len(tokenizer) == 65536

Tested with Transformers 4.57.6 / Tokenizers 0.22.2 and Transformers 5.16.1 / Tokenizers 0.23.2. No custom tokenizer code, remote-code trust or model weights are required. Pin a Hub commit with revision= for reproducible experiments.

The backend can also load tokenizer.json directly with tokenizers.Tokenizer.from_file.

Text and chat behavior

Whitespace, case and Unicode composition are preserved. There is no Unicode normalization, and the alphabet covers all 256 byte values. Digits are pretokenized individually. Some tokens contain only part of a UTF-8 character: decode complete sequences rather than concatenating separately decoded tokens.

Raw encoding does not automatically insert BOS/EOS: there is no tokenizer postprocessor. The chat template inserts the protocol boundaries explicitly:

messages = [{"role": "user", "content": "Kaj pomeni beseda prijateljstvo?"}]
ids = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=False
)

The template accepts string-valued system, user, assistant and tool messages. Structured multimodal inputs and native Gemma channel/tool protocols need separate integration. Control markers are reserved syntax; applications should handle their occurrence in untrusted message content deliberately.

A complete conversation ends with ⸢/s⸥; individual turns end with ⸢/TURN⸥. The supplied generation defaults use BOS 1, PAD 6, and EOS [2, 12]. Integrate them into the adapted model's configuration; loading the tokenizer does not change a model's stopping behavior. The maximum-length sentinel does not define a model context window.

Special-token contract

All special tokens use ⸢ (U+2E22) and ⸥ (U+2E25). The first 11 Micka IDs are preserved:

0 UNK, 1 s, 2 /s, 3 |, 4 p, 5 /p, 6 PAD, 7 MSK, 8 AGT, 9 USR, 10 SYS — each wrapped in the delimiters, for example ⸢s⸥.

See special_token_schema.json for every spelling and ID. Language, dictionary, tool and modality markers reserve an interface; their presence does not establish learned capabilities.

Training and vocabulary hygiene

Training-byte shares: 73.9412% Slovenian, 18.7317% English, 1.6285% Croatian, 0.8125% Bosnian, 1.6285% Serbian Latin and 3.2574% mixed Slovenian/code. Source labels supply the language accounting; independent language identification was not performed.

Slovenian inputs include web text, PDFs, translated mathematics, translated instruction text, parallel MaCoCu text, DGT translation memory and GaMS instructions. The release uses bounded samples of selected shards, not exhaustive or globally randomized corpus coverage. training_sources.json records contributions, revisions and source-file hashes. Only its 15 listed lanes were used; historical candidates with additional sources are separate artifacts.

A byte-level BPE candidate with 66,048 entries was trained locally, using minimum frequency 3 and maximum token length 64. A reviewed filter retained the first 65,152 admissible, dependency-valid learned merges, in rank order, alongside 256 byte tokens and 128 special tokens. Replacement entries come from lower-ranked learned merges, not synthetic filler.

The audit excluded 20 atomic source/agency/scrape entries, including questglobalpropsndocs, AddThis, OdgovoriIzbriši, LoveCoupons, DiRROS, Reuters, and Getty. STA and STA are retained, as valid uppercase forms of Slovenian sta, following the author's explicit instruction. (STA) consists of separate punctuation and word tokens, not a dedicated source marker.

The filter policy and complete exclusion audit are published. All strings remain encodable as smaller pieces. The filter changes vocabulary allocation; it does not remove those strings from the training text. No entry on the reviewed denylist remains in the vocabulary. This finite audit cannot certify that every possible obscure source-specific fragment is absent. Web/SEO noise and translated-text bias remain corpus limitations.

Independent evaluation

The final artifact was evaluated on 1,012 FLORES-200 devtest sentences in each of 23 language/script configurations, plus 500 new FineWeb-2 test documents each for Slovenian and Croatian. Web documents were selected with a seeded shuffle from one pinned test shard per language and restricted to 200–20,000 characters. This evaluation is separate from the tokenizer builder's original validation split.

Samples were checked against 676,595 unique normalized whole-document hashes from both historical training/validation corpora; no matching samples were found. This does not establish absence of substring, near-duplicate, shared-domain or semantic contamination. FLORES contains parallel translated sentences, so these results are not a universal estimate of web or conversational traffic. The multilingual training mix was not tuned on this test.

Each tokenizer receives identical text. Negative percentages mean fewer tokens than Gemma. Native GaMS3/Gemma 3 and Gemma 4 produced identical token counts on these samples. P95 is the 95th percentile of the per-sentence Micka/Gemma token-count ratio.

On FineWeb-2 test documents, Slovenian uses 30.2% fewer tokens and Croatian 13.4% fewer than Gemma 4. All 24,276 evaluated texts reconstruct exactly, with zero UNK. evaluation.json contains totals, per-character/byte metrics, sentence ratios, input hashes and donor identities. Only statistics and selection indices/hashes are published, not test text.

Interpretation: a sensible Slovenian-focused compromise, with useful Croatian compression and an approximately 12% English token penalty on the parallel test. It is not an efficient general-European tokenizer: German needs about 60% more tokens, Ukrainian 150% more, Greek 175% more, and Russian 191% more. Byte coverage provides representability, not multilingual understanding. Token counts do not measure model accuracy, perplexity or inference throughput.

Using with Gemma or GaMS

This is not a drop-in replacement for a pretrained model's tokenizer. Ordinary IDs differ from native Gemma/GaMS and earlier Micka variants. Adoption requires input-embedding/output-head remapping, retokenized training data, model configuration changes, adaptation training and held-out model evaluation.

donor_control_aliases.json records semantic control correspondences; native Gemma spellings are not automatically rewritten. Transplantation must map BOS/EOS/PAD/turn/modality controls by meaning before handling ordinary tokens. Ordinary byte-level pieces must use their decoded surfaces or underlying bytes, never their visible byte-alphabet spelling.

The donor audit found 23,381/65,408 ordinary entries (35.75%) initialize from a single native donor token; the remainder require multiple donor tokens. Gemma 4 supplies mappings for 27 reserved entries; 101 require fallback initialization. This is initialization evidence only: no donor model weights were transplanted or evaluated in this release. Modality markers alone do not establish processor or multimodal-model compatibility.

Usage statistics

The Hugging Face page exposes its standard public downloads and likes counters. For a timestamped JSON snapshot:

pip install huggingface_hub
python usage_stats.py

These are Hub-side counters, not unique users or tokenization calls. Hugging Face normally counts selected configuration/weight-file requests; tokenizer-only loads can be undercounted. See the Hub documentation. No client-side analytics, callbacks, usage logging or tracking code is installed by this tokenizer.

Identity and verification

Tokenizer SHA-256: b3326f8fc365e8f3f4b9545f6cafba8e2dab454352d825066c78f1384bbc6353.

The public name Micka-65k rounds the actual size of 65,536. This is the final filtered release, derived from the permissive-source corpus, and differs from the historical local “Micka-64k” candidates. See training_receipt.json, SHA256SUMS, and run python verify_tokenizer.py klokedm/Micka-65k --revision COMMIT_SHA for package checks.