Downloads · 30 days
0
ksopyla/minipile-unigram-32k-100k
minipile-unigram-32k-100k is a machine learning model from ksopyla. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is a Unigram tokenizer (SentencePiece-style) trained on the JeanKaddour/minipile dataset.
Downloads · 30 days
0
Access
Public
Updated Dec 2, 2025
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json2.1 MB · 100%
From the Hugging Face model README
This is a Unigram tokenizer (SentencePiece-style) trained on the JeanKaddour/minipile dataset.
Language: English (en). Note: While the Unigram algorithm handles unicode characters, the vocabulary is optimized for English text, code, and common technical terms found in Minipile.
It was developed for the MrCogito project, which explores novel transformer architectures like the Concept Encoder.
This tokenizer is specifically optimized for Concept Encoder and Concept Decoder architectures (e.g., Perceiver IO, Latent Transformers).
Concept Encoders work by compressing a sequence of tokens $T$ into a smaller set of abstract latent vectors (concepts) $C$. The efficiency of this compression depends heavily on the input quality:
unbe + liev + able). A Concept Encoder must waste model capacity "repairing" these arbitrary splits to understand the word before it can even begin extracting the higher-level concept.un + believ + able). This acts as a "soft pre-compression", feeding the encoder units that already carry semantic weight. This allows the Concept Encoder to focus its limited latent capacity on semantic aggregation rather than morphological repair.While XLNet uses the Unigram algorithm, its vocabulary (circa 2019) lacks modern terms. By training a fresh Unigram model on Minipile (2023), we combine the superior morphological segmentation of Unigram with the vocabulary coverage of modern LLMs (code, Python, ChatML, technical terms).
<|im_start|>, <|im_end|>) and a pre-configured chat template.from transformers import AutoTokenizer
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("ksopyla/minipile-unigram-32k-100k")
# Basic encoding
text = "Hello, world! This is a test."
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: [' Hello', ',', ' world', '!', ' This', ' is', ' a', ' test', '.']
# Chat Template
messages = [
{"role": "user", "content": "What is the Concept Encoder?"},
{"role": "assistant", "content": "It is a novel transformer architecture..."}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompt)
# Output: <|im_start|>user\nWhat is the Concept Encoder?<|im_end|>\n<|im_start|>assistant\n...
<pad><unk><cls><sep><mask><|im_start|>, <|im_end|>, <|user|>, <|assistant|>, <|system|>, <|endoftext|><|unused0|> ... <|unused99|> (100 reserved tokens)