Downloads · 30 days
0
mamba-tr-project-tokenizer
oguzatas/mamba-tr-project-tokenizer
mamba-tr-project-tokenizer is a machine learning model from oguzatas. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
This is a custom Byte-Pair Encoding (BPE) tokenizer trained specifically for the Turkish language to support the mamba-tr-project research.
Access
Public
Updated Apr 20, 2026
Repo size
282 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.4 MB · 89%
At a glance
- License
- mit
- Access
- Public
- Created
- Dec 2, 2025
- Updated
- Apr 20, 2026
- SHA
- 23e4a91d
Model card
From the Hugging Face model README
Turkish BPE Tokenizer for Mamba & GPT Models
This is a custom Byte-Pair Encoding (BPE) tokenizer trained specifically for the Turkish language to support the mamba-tr-project research.
Model Details
- Vocabulary Size: 32,000 tokens
- Training Corpus: Turkish Wikipedia (2023 Dump)
- Algorithm: BPE (Byte-Level)
- Special Tokens:
<|endoftext|>,<|padding|>
Usage
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained("oguzatas/mamba-tr-project-tokenizer")
text = "Yapay zeka günümüzde çok gelişti."
tokens = tokenizer.encode(text)
decoded = tokenizer.decode(tokens)
print(tokens)
# Output: [Token IDs...]
print(decoded)
# Output: "Yapay zeka günümüzde çok gelişti."
- License
- mit
- Languages
- tr
- Created
- Dec 2, 2025
- Updated
- Apr 20, 2026