my-tokenizer-turkish

My Tokenizer Turkish

This repository contains a small byte-level BPE tokenizer trained on a Turkish Wikipedia text corpus.

It is designed as a lightweight, general-purpose tokenizer for Turkish text. The tokenizer is packaged in the Hugging Face format so it can be loaded with AutoTokenizer or PreTrainedTokenizerFast.

Dataset

The tokenizer was trained on a plain-text .txt corpus included in this repository.

• Source file: the .txt training corpus stored alongside the tokenizer files
• Format: UTF-8 plain text
• Content: Turkish Wikipedia text

Keeping the raw TXT file in the same Hugging Face repository makes the training data easy to inspect and reproduce.

Tokenizer details

• Tokenizer type: Byte-level BPE
• Training data: the repository's plain-text .txt corpus
• Vocabulary size: 32,768
• Special tokens: <unk>, <pad>, <bos>, <eos>
• Pre-tokenization: ByteLevel
• Decoder: ByteLevel

Intended use

This tokenizer is intended for Turkish language experiments, especially when a compact subword tokenizer is preferred over a character-level vocabulary.

It works well as a general tokenizer for downstream language modeling, text generation, and other NLP tasks on Turkish text.

Training notes

The tokenizer was trained with the tokenizers library using a BPE model and the full byte-level alphabet. This makes the tokenizer robust to arbitrary input text, including punctuation, symbols, and unseen character sequences.

Loading the tokenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bilalabic/my-turkish-tokenizer")

text = "Merhaba, nasılsın?"
token_ids = tokenizer.encode(text)
tokens = tokenizer.convert_ids_to_tokens(token_ids)

print(tokens)
print(token_ids)
print(tokenizer.decode(token_ids))

Limitations

• This is a tokenizer only; it is not a language model.
• Quality depends on the training corpus used to build the vocabulary.
• It is trained for Turkish text, so performance on other languages may be weaker.

Files included

• tokenizer.json
• tokenizer_config.json

License

MIT