Downloads · 30 days
0
tahaenesaslanturk/ts-corpus-wordpiece-32k-uncased
ts-corpus-wordpiece-32k-uncased is a machine learning model from tahaenesaslanturk. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This repository contains a WordPiece tokenizer with a vocabulary size of 32,000, trained uncased on various datasets from the TS Corpus website. It is designed to handle Turkish text, leveraging rich and diverse sourc…
Downloads · 30 days
0
Access
Public
Updated Apr 17, 2024
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json767 KB · 99%
From the Hugging Face model README
This repository contains a WordPiece tokenizer with a vocabulary size of 32,000, trained uncased on various datasets from the TS Corpus website. It is designed to handle Turkish text, leveraging rich and diverse sources to provide a robust tool for natural language processing tasks.
The tokenizer was trained using multiple corpora from the TS Corpus, specifically:
These diverse sources include a wide range of texts from encyclopedic articles to legal documents, providing a comprehensive linguistic foundation for the tokenizer.
The tokenizer uses the WordPiece model, which is widely utilized in many modern NLP systems. It is particularly effective in handling languages with rich morphology like Turkish due to its subword segmentation approach. This tokenizer does not differentiate between uppercase and lowercase letters, ensuring uniformity in tokenization regardless of text casing.
To use this tokenizer, you can load it via the Hugging Face transformers library as follows:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("tahaenesaslanturk/ts-corpus-wordpiece-32k-uncased")