Downloads · 30 days
0
russian-unigram-20k
SherifAnar/russian-unigram-20k
russian-unigram-20k is a machine learning model from SherifAnar. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
50k+ слов с ria.ru, lenta.ru и др. (2020–2025)
Access
Public
Updated Oct 10, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json596 KB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Oct 9, 2025
- Updated
- Oct 10, 2025
- SHA
- 81729340
Model card
From the Hugging Face model README
Russian UNIGRAM Tokenizer 20000
🗃️ Корпус
50k+ слов с ria.ru, lenta.ru и др. (2020–2025)
⚙️ Параметры
- Алгоритм: UNIGRAM
- Размер словаря: 20,000
- Min frequency: 2
📊 Метрики
- OOV rate: 1.5%
- Reconstruction accuracy: 99.5%
- Compression ratio: 1.45
💻 Пример использования
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("SherifAnar/russian-unigram-20k")
text = "Привет, как дела?"
tokens = tokenizer.tokenize(text)
print(tokens) # ['П', 'ри', 'вет', ',', 'как', 'дела', '?']
📜 Лицензия
MIT
- License
- mit
- Languages
- ru
- Created
- Oct 9, 2025
- Updated
- Oct 10, 2025