nepali-e5-base-v1

nepali-e5-base-v1

A 278M-parameter, 768-dimensional retrieval embedding model for Nepali, fine-tuned from intfloat/multilingual-e5-base.

Built because there was no Nepali-focused retrieval embedding model. The Nepali models that existed on the Hub were sentence-similarity models, and general multilingual retrievers of comparable quality were twice the size.

Two things it does that nothing else in its class does:

1. On a 50,000-passage Nepali retrieval benchmark it matches bge-m3 and multilingual-e5-large — models with twice the parameters and 1024-dim vectors — at 2.4-4× their indexing throughput.
2. It is the only model tested with any functional handling of romanized Nepali (Nepali typed in Latin script). Every off-the-shelf multilingual retriever scores at or near chance there.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Ganu000/nepali-e5-base-v1")

# The "query: " and "passage: " prefixes are REQUIRED -- they are inherited from e5 and were
# used during fine-tuning. Omitting them costs roughly 10 nDCG points.
queries = model.encode(["query: सगरमाथा कति अग्लो छ?"])
passages = model.encode(["passage: सगरमाथा विश्वको सर्वोच्च शिखर हो। यसको उचाइ ८,८४९ मिटर छ।"])

similarity = model.similarity(queries, passages)   # cosine

• Output: 768-dimensional, cosine similarity
• max_seq_length: 256 tokens (~600 Devanagari characters). Longer input is silently truncated — chunk your documents below this.
• Scripts: Devanagari Nepali and English are both well supported. Romanized Nepali works partially — see the section below before relying on it.

Script support

Benchmark

92 hand-written natural Nepali questions (e.g. कञ्चनजङ्घा कुन देशको सीमामा छ?, सुगौली सन्धि कहिले भएको थियो?) against a 50,000-passage corpus drawn from Nepali Wikipedia. Every model was evaluated on the identical corpus, capped at the same 256 tokens, with each model's own required prefixes or prompts applied.

The 90 Wikipedia articles supplying gold answers were excluded from this model's training data. Gold articles were excluded whole, not just their lead paragraph, because a later paragraph of the right article frequently also answers the question.

Devanagari queries (50,000-passage corpus)

docs/s is batch indexing throughput on an RTX 4050 laptop at 256 tokens.

Romanized queries (same corpus, same gold passages)

Every off-the-shelf model is non-functional here. On a 50,092-passage corpus, 0-8 hits out of 92 is at or near chance. None of them has seen romanized Nepali in any form.

⚠️ Read 0.2983 as a ceiling, not a field measurement. The benchmark's romanized queries were produced by the same rule-based transliterator that generated the training anchors, so the model is being tested on spellings it was explicitly taught. Real users spell inconsistently and apply medial schwa deletion the transliterator does not (sagarmatha, not sagaramatha). Performance on human-typed romanized Nepali is unverified and will be lower.

If romanized input matters to your application, the stronger architecture is to transliterate Roman → Devanagari at query time (e.g. with ai4bharat's IndicXlit) and use the Devanagari path, which scores 0.6373. All of the model's pretrained knowledge lives in Devanagari.

What the numbers say

Tied with the 560M/1024-dim models. −2.2 nDCG against e5-large, +0.9 against bge-m3. Both inside the ~4-point noise floor of a 92-query benchmark. Same quality, 2.4-4× the indexing throughput, 25% smaller vectors.

Best Recall@5 and Recall@10 in the field, though by 2.2 points each — real but marginal. If you rerank, Recall@10 is the metric that matters: a reranker can reorder what you retrieved but cannot recover what you missed.

Ahead of every Nepali-specific model. +5.0 nDCG over the best of them. Note that those models' training corpora are undisclosed; since Nepali Wikipedia is the dominant Nepali corpus, they may have trained on the articles this benchmark holds out. The comparison is, if anything, tilted against this model.

Retrieval training beats language training. Untuned multilingual-e5-base, with no Nepali adaptation whatsoever, already matches the best Nepali-specific model. The two sentence_similarity_nepali models score near zero — not because they are bad, but because CosineSimilarityLoss on STS data teaches symmetric similarity, not asymmetric retrieval. Wrong task, not wrong model.

Corpus size changes the answer

The same models on a 4,092-passage version of this benchmark:

Every score drops ~20 points and the ranking inverts. A small evaluation corpus flatters every model and can reverse conclusions. Always report corpus size alongside a retrieval score, and treat even 50k as optimistic relative to a production corpus of millions.

Training

query: and passage: prefixes were applied to anchors and positives during training, so they are mandatory at inference.

The 6 GB VRAM budget drove three choices: gradient-cached MNRL (so a 256-pair batch of in-batch negatives fits with only 4 sentences resident), Adafactor instead of AdamW (whose two fp32 moments alone need ~2.2 GB for 278M parameters), and bf16 autocast.

Data

Romanization was applied to anchors only, never to passages — users type queries in Latin script while documents stay in Devanagari — and only to anchors under 200 characters, since nobody types a paragraph into a search box.

21,080 Nepali Wikipedia articles contributed, out of 32,885 in the 20231101.ne dump — 10,124 were stubs with no paragraph over 120 characters and 791 were list or meta pages. OPUS-100 required ~406,000 streamed rows to yield 52,404 usable unique pairs: 264,094 failed a length-ratio and script filter, and roughly 90,000 more were exact duplicates.

No human labelling. MultipleNegativesRankingLoss needs only positive pairs, with negatives taken from the batch.

Limitations

Wikipedia domain. Training and evaluation are both Nepali Wikipedia — same register, same length, same style. Expect lower numbers on a different domain. Build a small evaluation set from your own documents before trusting these figures in production.

92 evaluation queries. Differences under about 4 nDCG points are not measurable. The confidence interval on any single score here is roughly ±9 points. More queries, not more training data, is the fix.

Romanized support is partial and measured against a synthetic transliteration. See the romanized benchmark section above. Treat 0.2983 as a ceiling.

Query shape mismatch. Training anchors are titles and opening sentences; real queries are questions. Transfer works, but templating declaratives into questions would likely close some of the Recall@1 gap to e5-large.

~4% of Nepali training passages are list-shaped Wikipedia stubs — bot-generated place-name lists rather than prose. Measured and left in, because the one-line fix would also change the evaluation set and break comparability with every number above.

Hard negatives did not help. A variant trained on mined hard-negative triplets scored statistically identically, trading Recall@1 gains for Recall@10 losses at no net benefit. Not published.

Only ~60k of the Devanagari pairs are Nepali-to-Nepali. The majority of the non-romanized signal is cross-lingual alignment, capped by Nepali Wikipedia having only 21k usable articles.

Reproducing

Write-up, full benchmark tables, and method: https://github.com/Ganu000/nepali-embeddings

The whole pipeline runs on a single 6 GB laptop GPU in about three hours including data preparation. The evaluation set is 113 hand-written Nepali questions mapped to Wikipedia articles, of which 92 resolve against the 20231101.ne dump.

Licensing and attribution

• Base model: intfloat/multilingual-e5-base, MIT.
• Nepali Wikipedia text: CC BY-SA 4.0. Attribution to Wikipedia contributors.
• OPUS-100: see the dataset card for its per-corpus licensing.

Weights are released under Apache 2.0. Whether CC BY-SA training data imposes downstream obligations on model weights is legally unsettled; confirm for your own use case.

Citation

@misc{nepali-e5-base-v1,
  title  = {nepali-e5-base-v1: a retrieval embedding model for Nepali},
  author = {Ganesh Sah},
  year   = {2026},
  url    = {https://huggingface.co/Ganu000/nepali-e5-base-v1}
}