qwen3-embedding-8b-tibetan-lora

Tibetan-aware retrieval LoRA for Qwen3-Embedding-8B

Qwen3-Embedding-8B cannot see Tibetan: a Sanskrit query retrieves the matching Wylie dictionary headword 4% of the time, an English query 0.4% of the time, and even Wylie ↔ Tibetan-script (uchen) — a deterministic transliteration — round-trips at only 43% R@1. Buddhist-text RAG systems work around this with hand-maintained alias tables.

This LoRA teaches the mapping to the embedder itself: Wylie ↔ uchen ↔ Sanskrit (IAST) ↔ English, trained contrastively on 95k alias pairs mined from a licensed copy of the Illuminator Tibetan-English Encyclopaedic Dictionary (©Tony Duff, Padma Karpo Translation Committee — the dictionary text itself is not included in this repo and cannot be reconstructed from embedding weights).

Results — dictionary headword retrieval (held-out terms)

R@1 over the full 26,787-headword haystack. Held-out terms (300 Sanskrit

• 500 English) contributed zero training pairs.

Results — passage retrieval (real practice-text corpus)

58 gold queries (14 dharma terms × up to 4 transliteration schemes + 4 end-to-end questions) against 32,607 passages (gold targets + 30k distractors) from a 866-document practice-text collection. hit@5 = any gold passage in top 5; cross-hit masks out every passage that contains the query string literally — the score string-matching cannot reach.

Read this table honestly: it is a specialization trade-off, measured on small n (12-14 queries per scheme). Tibetan-side retrieval jumps — cross-hit (the case alias tables exist for) goes 4/14→11/14 on Wylie and 2/14→10/14 on phonetic. Phonetic renderings were learned transitively: the dictionary contains no phonetics at all, yet "tongpa nyi" now finds stong pa nyid passages. The price: general-semantic English retrieval regresses (5/14→2/14, end-to-end questions 0/4) — the LoRA pulls English glosses toward Tibetan headword space and away from general passage space. Dictionary-level English→headword improved (0.004→0.186), so the regression is specific to long-passage semantic matching.

Deployment recommendation: route queries. Adapter ON for Tibetan-script/Wylie/phonetic/Sanskrit-term queries, adapter OFF (base model) for general English questions — with PEFT this is enable_adapter_layers()/disable_adapter_layers() per query, no extra memory. In a hybrid stack (FTS + vector + reranker) the English regression is further absorbed by the lexical side.

Usage

Exactly like base Qwen3-Embedding-8B (last-token pooling, L2-normalize, instruction prefix on queries only) with the adapter attached:

from transformers import AutoModel, AutoTokenizer
from peft import PeftModel
import torch, torch.nn.functional as F

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-Embedding-8B", padding_side="right")
model = AutoModel.from_pretrained("Qwen/Qwen3-Embedding-8B", torch_dtype=torch.bfloat16).cuda()
model = PeftModel.from_pretrained(model, "anicka/qwen3-embedding-8b-tibetan-lora").eval()

INSTRUCT = ("Instruct: Given a term in any transliteration scheme "
            "(Wylie, Tibetan script, Sanskrit, English), retrieve the "
            "matching Tibetan dictionary headword\nQuery: ")

def embed(texts, prefix=""):
    enc = tok([prefix + t for t in texts], padding=True, truncation=True,
              max_length=256, return_tensors="pt").to("cuda")
    h = model(**enc).last_hidden_state
    idx = enc.attention_mask.sum(1) - 1
    v = h[torch.arange(h.size(0)), idx]
    return F.normalize(v.float(), dim=-1)

q = embed(["tongpa nyi"], prefix=INSTRUCT)          # phonetic query
d = embed(["stong pa nyid", "སྟོང་པ་ཉིད་", "śūnyatā"])  # any scheme
print(q @ d.T)

Training

• Data: 95,135 contrastive pairs from 26,787 dictionary entries: wylie↔uchen (26k), english-gloss→wylie (36k), form→definition (22k), abbreviation/misspelling→canonical (3.6k), sanskrit→wylie/uchen (4k), verb-stem→lemma (3.2k). Held-out eval terms excluded from all pair types.
• Objective: symmetric InfoNCE, in-batch negatives, temperature 0.05. Length-bucketed batches (terms vs definitions) — term batches give all-term in-batch negatives, which are the hard ones.
• LoRA: r=16, α=32, all attention + MLP projections, 1 epoch (best val at end, no overfit), lr 1e-4 cosine, batch 48, bf16 + gradient checkpointing.
• Trained with the query instruction prefix on anchors only, matching inference usage.

Limitations

• General-English and question-style retrieval regress (see the passage table above) — route those to the base model. The natural v2 fix is mixing general-retrieval pairs into training to pin the English side in place.
• English glosses collide ("wisdom" maps to shes rab, ye shes, …); english-axis R@1 undercounts real quality — check R@5.
• Trained on dharma vocabulary; no claim about secular modern Tibetan.
• Passage-eval n is small (12-14 queries/scheme); treat those fractions as strong signal with wide error bars, not precise rates.

Acknowledgements

The Illuminator Tibetan-English Encyclopaedic Dictionary (Lotsawa Tony Duff, Padma Karpo Translation Committee) — used under license as training data. Motivated by a Buddhist-RAG builder's observation that no neural embedder handles Tibetan transliteration schemes.