Downloads · 30 days
0
russian-bpe-tokenizer
kandrey01/russian-bpe-tokenizer
russian-bpe-tokenizer is a machine learning model from kandrey01. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Access
Public
Updated Oct 11, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json618 KB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Oct 10, 2025
- Updated
- Oct 11, 2025
- SHA
- c5832399
Model card
From the Hugging Face model README
[Russian BPE Tokenizer 16k]
🗃️ Корпус
50k+ слов с rbc.ru
⚙️ Параметры
- Алгоритм: BPE
- Размер словаря: 16,000
- Min frequency: 2
📊 Метрики
- OOV rate: 2.4%
- Reconstruction accuracy: 82.5%
- Compression ratio: 2.14
💻 Пример использования
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("kandrey01/russian-bpe-tokenizer")
tokens = tokenizer.tokenize("Привет, как дела?")
- License
- mit
- Created
- Oct 10, 2025
- Updated
- Oct 11, 2025