Downloads · 30 days
0
russian-bpe-16k
SherifAnar/russian-bpe-16k
russian-bpe-16k is a machine learning model from SherifAnar. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
50k+ слов с ria.ru, lenta.ru и др. (2020–2025)
Access
Public
Updated Oct 10, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1 MB · 84%
At a glance
- License
- mit
- Access
- Public
- Created
- Oct 10, 2025
- Updated
- Oct 10, 2025
- SHA
- 41b81dc1
Model card
From the Hugging Face model README
Russian BPE Tokenizer 16000
🗃️ Корпус
50k+ слов с ria.ru, lenta.ru и др. (2020–2025)
⚙️ Параметры
- Алгоритм: BPE
- Размер словаря: 16,000
- Min frequency: 2
📊 Метрики
- OOV rate: 1.2%
- Reconstruction accuracy: 99.8%
- Compression ratio: 1.35
💻 Пример использования
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("SherifAnar/russian-bpe-16k")
text = "Привет, как дела?"
tokens = tokenizer.tokenize(text)
print(tokens) # ['ри', 'вет', ',', 'как', 'дела', '?']
📜 Лицензия
MIT
- License
- mit
- Languages
- ru
- Created
- Oct 10, 2025
- Updated
- Oct 10, 2025