turkish-books-bpe-tokenizer

Model Card for Turkish Books BPE Tokenizer

<!-- Provide a quick summary of what the model is/does. -->

This model is a BPE tokenizer obtained from the concatenated version of Turkish Book Data Set.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Filiz Yalçin
• Model type: BPE Tokenizer
• Language(s) (NLP): Turkish (tr)
• License: gpl-3.0

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

This tokenizer can be used to tokenized Turkish texts.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer

# 1. Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("filiz-yalcin/turkish-books-bpe-tokenizer")

# 2. Encode text to IDs
encoded = tokenizer("Yalnızlık paylaşılmaz. Paylaşılsa yalnızlık olmaz.")
print(encoded["input_ids"])

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The corpus was obtained using Turkish Book Data Set. The 'explanation' column's values were concatenated.

<!-- ### Training Procedure --> <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->