Bo-Yigchung-Detection

mmBERT Yigchung

Token classifier for Yigchung in Classical Tibetan: the small-print interlinear annotation written under the main line. The model reads a flat base string and returns character spans. It does not see page images.

Fine-tune of jhu-clsp/mmBERT-base (ModernBertForTokenClassification, 22 layers, max length 8192) on karma689/yigchung_detection, revision v1.0.

This is a new experiment line (232 books, 3-class BIO).

How to run inference

Spans are inclusive on both ends: text[start:end + 1].

Books are longer than the 8192-token window. Cut the same windows used in training, keep each content token from the first window that covers it, then Viterbi-decode. break_penalty=1 is the value chosen on the validation split.

pip install "transformers>=4.40" torch numpy
git clone https://github.com/Webuddhist-tech/tibetan-yigchung-detection
cd tibetan-yigchung-detection
python src/script/infer_yigchung.py --text-file book.txt

JSON rows look like {"start": 1204, "end": 1388, "text": "..."}.

From Python, after the repo is on PYTHONPATH (run from the clone, or sys.path includes it):

from src.script.infer_yigchung import predict_spans

text = open("book.txt", encoding="utf-8").read()
for start, end in predict_spans(text):
    print(start, end, text[start:end + 1])

predict_spans loads karma689/mmbert-yigchung, uses CUDA when it is available, and defaults to break_penalty=1 and batch size 2. Override with predict_spans(text, device="cpu", break_penalty=1.0).

What the script is doing

A span longer than 8190 content tokens cannot sit inside one window. The longest span in this corpus is 4,823 tokens, so the 40% overlap covers every gold span without rescue windows. Rescue windows were not used at train or test time for this geometry.

Results

Greedy one-to-one span match at IoU ≥ 0.5. Checkpoint is step 2768 (epoch 4.0), the best validation [email protected] F1 during training (in-loop penalty 2.0, F1 0.665). The penalty was then re-chosen on that checkpoint and the test set was scored once.

Validation F1 by break_penalty on this checkpoint: 0 → 0.659, 1 → 0.666, 2 → 0.665, 4 → 0.665, 8 → 0.655.

The test drop against validation is real. A few test books contribute many false negatives (one book has 107 gold spans and no true positives at penalty 1). Treat 0.431 as the number to cite for this release.

Training

Train / validation / test windows: 5,546 / 696 / 722. About 78% of train windows contain no Yigchung token. Overlapped tokens are labelled in exactly one window.

Data

232 reviewed Classical Tibetan books, 8,495 Yigchung spans, 51,535,768 characters. Book-level 80/10/10 split (184 / 24 / 24). A book never appears in two splits. Splits were stratified by catalog family (P / I) and span-count bin, then swapped inside a stratum until span density matched (coverage ratio 1.00× across splits).

Gold config on the Hub is the book text plus spans. The windowed config windowed_w8192_s4914 is derived from tokenizer jhu-clsp/mmBERT-base and this geometry. Do not mix it with windows from another tokenizer or stride.

Label scheme

The published windows are already these three ids. Do not remap them from a 15-class head. Label -100 is the loss mask for specials and for tokens already owned by an earlier window; it is not a class.

License and attribution

Base model license follows jhu-clsp/mmBERT-base (Apache-2.0). Source texts were digitized and made available by the Buddhist Digital Resource Center (BDRC). Annotations were prepared through OpenPecha with support from the Tsadra Foundation.