turkish-bpe-tokenizer-128

Turkish BPE Tokenizer (128k Vocabulary)

This repository contains a Byte-Level BPE (Byte Pair Encoding) tokenizer specifically trained on a Turkish corpus (Uunan/turkish-synthetic-corpus). It is designed to optimally tokenize Turkish texts with a vocabulary size of 128,000 tokens.

Tokenizer Details

• Tokenizer Type: Byte-Level BPE
• Vocabulary Size: 128,000
• Normalizer: NFC (Normalization Form C)
• Special Tokens:
  • <pad>
  • <unk>
  • <bos>
  • <eos>
• Post-Processor: Template processing for <bos> and <eos> tokens.

How to Use

You can easily use this tokenizer using the transformers library or the tokenizers library.

Using transformers

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
tokenizer.pad_token = "<pad>"
tokenizer.unk_token = "<unk>"
tokenizer.bos_token = "<bos>"
tokenizer.eos_token = "<eos>"

text = "Türkiye yapay zekâ alanında büyük yatırımlar yapıyor."
encoded = tokenizer.encode(text)
print(encoded)

decoded = tokenizer.decode(encoded)
print(decoded)

Using tokenizers

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")

text = "Türkiye yapay zekâ alanında büyük yatırımlar yapıyor."
encoded = tokenizer.encode(text)

print("Tokens:", encoded.tokens)
print("IDs:", encoded.ids)

Training Data

This tokenizer was trained on the Uunan/turkish-synthetic-corpus dataset, which contains high-quality Turkish synthetic text data, ensuring better coverage for modern Turkish language semantics and morphological structures.

Performance

With a vocabulary size of 128k, this tokenizer minimizes the sequence length for Turkish text generation and understanding tasks, improving the efficiency of Language Models (LLMs) trained on Turkish.