turkishwkpd-tokenizer

Model Card for Model ID

This is a Byte-Pair Encoding (BPE) tokenizer trained from scratch on Turkish Wikipedia text using the 🤗 Hugging Face tokenizers library. It is intended for use as the tokenizer component of Turkish-language NLP models.

Model Details

Model Description

This is a Byte-Pair Encoding (BPE) tokenizer trained from scratch on Turkish Wikipedia text using the 🤗 Hugging Face tokenizers library. It is intended for use as the tokenizer component of Turkish-language NLP models.

• Developed by: Talayhan
• Model type: BPE Tokenizer (vocabulary size: 8,192)
• Language(s) (NLP): Turkish (tr)

Model Sources

• Repository: Trained using the train_hf_tokenizer.py script from malibayram/single_letter_transformers

Uses

Direct Use

This tokenizer can be used to convert Turkish text into subword tokens (BPE) for downstream use in training or fine-tuning Turkish NLP models (e.g., language models, classifiers).

Out-of-Scope Use

Not recommended for languages other than Turkish, or for domains substantially different from Wikipedia text (e.g., code, social media slang, dialectal variants), where the vocabulary coverage may be poor.

Bias, Risks, and Limitations

As the tokenizer was trained solely on Turkish Wikipedia, it may underrepresent informal language, dialectal variation, code-switching, and domain-specific vocabulary (e.g., medical, legal, colloquial). A vocabulary size of 8,192 is relatively small, which may lead to more aggressive subword splitting for rare or out-of-domain words.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the tokenizer. Consider evaluating tokenization quality (fertility, coverage) on your target domain before use, especially if it differs from encyclopedic text.

Training Details

Training Data

Turkish Wikipedia dump, sourced from kaan39/turkish-wikipedia-dataset on the Hugging Face Hub. The dataset contains 26,871 rows with a total file size of 37.7 MB.

Training Procedure

Preprocessing

Only the messages (text content) field was extracted from the dataset and written to a plain .txt file; the tokenizer was trained on this extracted text.

Training Hyperparameters

• Algorithm: Byte-Pair Encoding (BPE)
• Vocabulary size: 8,192 (2^13)

Technical Specifications

Model Architecture and Objective

BPE (Byte-Pair Encoding) subword tokenizer, vocabulary size 8,192, trained with the Hugging Face tokenizers library.