Downloads · 30 days
0
tahaenesaslanturk/ts-corpus-wordpiece-256k-cased
ts-corpus-wordpiece-256k-cased is a machine learning model from tahaenesaslanturk. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This repository contains a WordPiece tokenizer with a vocabulary size of 256,000, trained uncased on various datasets from the TS Corpus website. It is designed to handle Turkish text, leveraging rich and diverse sour…
Downloads · 30 days
0
Access
Public
Updated Apr 30, 2024
Repo size
—
Likes
1
Public
Click a slice to open those files.
.json6.8 MB · 100%
From the Hugging Face model README
This repository contains a WordPiece tokenizer with a vocabulary size of 256,000, trained uncased on various datasets from the TS Corpus website. It is designed to handle Turkish text, leveraging rich and diverse sources to provide a robust tool for natural language processing tasks.
The tokenizer was trained using multiple corpora from the TS Corpus, specifically:
These diverse sources include a wide range of texts from encyclopedic articles to legal documents, providing a comprehensive linguistic foundation for the tokenizer.
The tokenizer uses the WordPiece model, which is widely utilized in many modern NLP systems. It is particularly effective in handling languages with rich morphology like Turkish due to its subword segmentation approach. This tokenizer differentiates between uppercase and lowercase letters.
To use this tokenizer, you can load it via the Hugging Face transformers library as follows:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("tahaenesaslanturk/ts-corpus-wordpiece-256k-cased")