Downloads · 30 days
0
Russian_BPE_Tokenizer_8k
IVIaximushka/Russian_BPE_Tokenizer_8k
Russian_BPE_Tokenizer_8k is a machine learning model from IVIaximushka. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Access
Public
Updated Sep 27, 2025
Repo size
—
Likes
1
Public
Repo makeup
Click a slice to open those files.
.json623 KB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Sep 27, 2025
- Updated
- Sep 27, 2025
- SHA
- 4e986492
Model card
From the Hugging Face model README
Название модели: Russian BPE Tokenizer 8k]
Корпус
50k+ слов с kommersant.ru. (2020–2025)
Параметры
- Алгоритм: BPE
- Размер словаря: 8,000
- Min frequency: 2
Метрики
- Reconstruction accuracy: 100%
- Compression ratio: 0.6972995985889795
Пример использования
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("saved_models/BPE_vocab8000_20250927_104253.json")
tokenizer.encode("я иду в школу").tokens
Лицензия
MIT
- License
- mit
- Languages
- ru
- Created
- Sep 27, 2025
- Updated
- Sep 27, 2025