xnli-morphel-zh-32k

NIRVLab — MorpheL Tokenizer for Chinese XNLI

MorpheL: MI-Guided Stochastic Segmentation tokenizer for morphologically rich low-resource languages (Proposal: MorpheL: Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource Languages).

Trained on the Chinese (zh) subset of facebook/xnli — all splits.

Algorithm

MorpheL scores candidate intra-word boundaries via pointwise mutual information (MI) between prefix and suffix substrings, then stochastically selects the number of cuts via Gumbel perturbation (Eq. 6–8).

Chinese-specific design choice: ZH_VOWELS = ∅ (empty set). The vowel-consonant transition heuristic (Proposal §5.1 Step 1) returns False for all positions in pure Hanzi text, so B(w) = ∅ for all Hanzi words → segment_word returns [word] (no cuts). This is correct: Chinese Hanzi do not have a Latin vowel/consonant distinction to exploit for morpheme boundary detection. The vocabulary is built from whole-token Hanzi sequence frequencies (32k slots), plus a character-level overflow alphabet for guaranteed <unk>-free encoding.

Training Config

Evaluation Metrics (vs Baselines, vocab_size=32000, same corpus)

Fill baseline columns after running baseline notebooks.

Usage

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-morphel-zh-32k")

Note: MorpheL segments words before passing to the tokenizer. For Chinese, segment_word returns [word] for all Hanzi tokens (no cuts). Downstream training uses temperature=1.0 for stochastic segmentation; inference uses temperature=0 (deterministic).