Downloads · 30 days
0
xnli-morphel-ru-32k
NIRVLab/xnli-morphel-ru-32k
xnli-morphel-ru-32k is a machine learning model from NIRVLab. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
MorpheL: MI-Guided Stochastic Segmentation tokenizer for morphologically rich low-resource languages (Proposal: MorpheL: Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource Languages).
Access
Public
Updated Jul 6, 2026
Repo size
12.3 MB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pkl12.3 MB · 87%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jul 6, 2026
- Updated
- Jul 6, 2026
- SHA
- 33f525c7
Datasets
Model card
From the Hugging Face model README
NIRVLab — MorpheL Tokenizer for Russian XNLI
MorpheL: MI-Guided Stochastic Segmentation tokenizer for morphologically rich low-resource languages (Proposal: MorpheL: Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource Languages).
Trained on the Russian (ru) subset of
facebook/xnli — all splits.
Algorithm
MorpheL scores candidate intra-word boundaries via pointwise mutual information (MI) between prefix and suffix substrings, then stochastically selects the number of cuts via Gumbel perturbation (Eq. 6–8). Key distinctions from BPE-Dropout:
- Randomness is MI-informed — only high-MI boundaries enter the candidate pool
- Vowel-consonant transition heuristic pre-filters linguistically implausible positions
- Global MI table aggregated over full corpus (not per-sentence) for stability
Training Config
| Parameter | Value |
|---|---|
| Algorithm | MorpheL (MI + Gumbel) |
| Vocabulary size | 32,083 |
| top_k | 4 |
| temperature (T) | 1.0 (vocab induction: T=0) |
| mi_threshold | 0.0 (keep MI > 0) |
| min_frequency | 2 |
| Special tokens | <s>, <pad>, </s>, <unk>, <mask> |
| Corpus | facebook/xnli/ru — all splits (800,404 sentences) |
| Vowel set | Russian Cyrillic (а е ё и о у ы э ю я — 10 vowel letters; й/ь/ъ excluded) |
Evaluation Metrics (vs Baselines, vocab_size=32000, same corpus)
| Metric | BPE | WordPiece | Unigram | MorpheL |
|---|---|---|---|---|
| Fertility ↓ | — | — | — | 1.4282 |
| Tokens/char ↓ | — | — | — | 0.2076 |
| Avg seq len ↓ | — | — | — | 19.27 |
| Vocab coverage ↑ | — | — | — | 1.0000 |
| OOV rate ↓ | — | — | — | 0.0000 |
Fill baseline columns after running baseline notebooks.
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-morphel-ru-32k")
Note: MorpheL segments words before passing to the tokenizer. At downstream training time, use
temperature=1.0for stochastic segmentation. For inference, usetemperature=0(deterministic).
- License
- apache-2.0
- Languages
- ru
- Created
- Jul 6, 2026
- Updated
- Jul 6, 2026