my-tokenizer

my-tokenizer

byte-level Byte-Pair Encoding (BPE) tokenizer trained from scratch with the tokenizers library. It was built as a learning exercise: text is scraped from a Wikipedia article, a BPE tokenizer is trained on it, and the result is wrapped as a PreTrainedTokenizerFast so it loads through AutoTokenizer.

Details

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gorkemergune/my-tokenizer")

example = "Hello! This is a very small tokenizer example."
token_ids = tokenizer.encode(example)

print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids))
print("Token IDs:", token_ids)
print("Decoded:", tokenizer.decode(token_ids))

Limitations

• Trained on a single, short Wikipedia article, so its merges are biased toward that domain (LLM / machine-learning vocabulary) and English text.
• The vocabulary is intentionally tiny (2048), so longer or rarer words are split into many sub-tokens. This is a demo tokenizer, not a production one.

How it was built

The training pipeline is available on GitHub (gorkemergune/wiki2bpe). In short:

1. scraper.py downloads the Wikipedia article as plain text into text.txt.
2. script.py trains the byte-level BPE tokenizer on text.txt and pushes it here.