xnli-morphel-tr-32k

NIRVLab — MorpheL Tokenizer for Turkish XNLI

MorpheL: MI-Guided Stochastic Segmentation tokenizer for morphologically rich low-resource languages (Proposal: MorpheL: Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource Languages).

Trained on the Turkish (tr) subset of facebook/xnli — all splits.

Algorithm

MorpheL scores candidate intra-word boundaries via pointwise mutual information (MI) between prefix and suffix substrings, then stochastically selects the number of cuts via Gumbel perturbation (Eq. 6–8). Key distinctions from BPE-Dropout:

• Randomness is MI-informed — only high-MI boundaries enter the candidate pool
• Vowel-consonant transition heuristic pre-filters linguistically implausible positions
• Global MI table aggregated over full corpus (not per-sentence) for stability

Training Config

Evaluation Metrics (vs Baselines, vocab_size=32000, same corpus)

Fill baseline columns after running baseline notebooks.

Usage

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-morphel-tr-32k")

Note: MorpheL segments words before passing to the tokenizer. At downstream training time, use temperature=1.0 for stochastic segmentation. For inference, use temperature=0 (deterministic).