Downloads · 30 days
0
cl100k_base-mlm
BEE-spoke-data/cl100k_base-mlm
cl100k_base-mlm is a machine learning model from BEE-spoke-data. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Access
Public
Updated Dec 29, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json5.8 MB · 86%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Feb 3, 2024
- Updated
- Dec 29, 2025
- SHA
- a8dfac9a
Model card
From the Hugging Face model README
tiktoken cl100k_base: as HF MLM tokenizer
based on RobertaTokenizerFast
from pathlib import Path
from transformers import RobertaTokenizerFast, AutoTokenizer
repo_id = "BEE-spoke-data/cl100k_base-mlm"
tk = AutoTokenizer.from_pretrained(repo_id)
len(tk)
# 100266
testing that it does what it should:
input_text = "i love memes"
tokenized_ids = tk.encode(input_text)
decoded_tokens = tk.convert_ids_to_tokens(tokenized_ids)
print(f"for input '{input_text}' -> {tokenized_ids} -> {decoded_tokens}")
# for input 'i love memes' -> [100277, 72, 3021, 62277, 100278] -> ['<s>', 'i', 'Ġlove', 'Ġmemes', '</s>']
- Library
- transformers
- License
- mit
- Created
- Feb 3, 2024
- Updated
- Dec 29, 2025