muril-hinglish-lid

muril-hinglish-lid

Context-aware Hindi/English language identification for romanized Hinglish.

Labels every word of code-mixed Latin-script text as HIN or ENG, using the surrounding sentence rather than a dictionary.

Example

Input:

Mujhe calculus ka doubt samajh nahi aaya, please explain.

Output:

Why context matters

The same string can be two different languages in one sentence:

Main   is    question  ka    main  point  samajh  nahi  paya
HIN    HIN   ENG       HIN   ENG   ENG    HIN     HIN   HIN
 └─ मैं  └─ इस                  └─ English "main"

Main is मैं; four words later main is the English adjective. is is इस, not the English copula. No lookup table can express this — it has one row per string.

That is not a contrived example. In our evaluation data the word to occurs 775 times, splitting 435 Hindi (तो) / 340 English. A perfect lookup table gets at most 56% of those by always guessing the majority. This model gets 98.32%.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

MODEL = "PhysicsWallahAI/muril-hinglish-lid"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForTokenClassification.from_pretrained(MODEL).eval()


def tag_words(words: list[str]) -> list[str]:
    """One label per word, read off the word's FIRST sub-token."""
    enc = tok(words, is_split_into_words=True, truncation=True,
              max_length=256, return_tensors="pt")
    with torch.no_grad():
        pred = model(**enc).logits[0].argmax(-1).tolist()
    out, prev = [], None
    for pos, wid in enumerate(enc.word_ids()):
        if wid is not None and wid != prev:
            out.append(model.config.id2label[pred[pos]])
        prev = wid
    return out


words = "Mujhe calculus ka doubt samajh nahi aaya please explain".split()
print(list(zip(words, tag_words(words))))
# [('Mujhe','HIN'), ('calculus','ENG'), ('ka','HIN'), ('doubt','ENG'),
#  ('samajh','HIN'), ('nahi','HIN'), ('aaya','HIN'), ('please','ENG'),
#  ('explain','ENG')]

Use this helper rather than pipeline(..., aggregation_strategy=...). The pipeline's aggregation is designed for NER: it merges consecutive tokens sharing a label into one span, which is right for New York City → one LOC and wrong here, where adjacent words routinely share a language. On the example above it returns 7 spans instead of 9 words, fusing samajh nahi aaya, into a single HIN blob. aggregation_strategy="none" is not the fix either — it returns sub-word pieces ('Mu', '##jhe').

The helper matches the training-time contract exactly: words pre-split, one label per word taken from its first sub-token. Continuation pieces were masked out of the loss during training and carry no supervision, so reading them — or averaging over them, which would let a long word's many pieces outvote a short word's one — asks the model something it was never taught to answer.

Tag in windows of ~100 words, which is the window used in training.

Model

• MuRIL backbone (google/muril-base-cased), pretrained on 17 Indian languages and their romanized forms
• Token-classification head, 2 labels
• 237 M parameters · 12 layers · hidden 768 · WordPiece vocab 197,285
• Labels: HIN / ENG
• Context-aware prediction

Performance

Held-out gold set

874 answers / 108,432 tokens. The evaluation protocol is the part worth reading: these labels were annotated from the source text by a separate model that wrote none of the training corpus. Training labels were derived by aligning Hinglish text to its Devanagari rewrite; the gold labels were not. So this is the only measurement here that is independent of the pipeline that produced the training data.

On its own auto-labelled test split the model reads 0.9934 — but that split is the instrument that cannot see its own errors. The gold figure is the one to quote.

Hard cases

Out of domain: LinCE Hindi-English

LinCE is human-annotated, public, and from a different domain entirely — code-mixed tweets, not tutoring text.

Not comparable to the LinCE leaderboard. LinCE has eight classes; this model has two. We map lang1→ENG, lang2→HIN and drop the rest — other (punctuation, handles, URLs, emoji; 2,231 tokens), ne (named entities; 875), and 37 tokens of fw/mixed/unk/ambiguous. Scoring only the two mappable classes is an easier task than LinCE's. Compare this number to our in-domain accuracy, and to nothing else.

Latency

0.099 ms/word (108 words in 10.7 ms), NVIDIA A10, fp32, batch 1.

Training

Tokens the aligner could not place — math variables, symbols, ambiguous residue — were emitted as O and masked out of the loss rather than guessed at, so the model was never trained to invent a label for something the labelling process did not actually know.

Intended use

• Hinglish language identification
• Preprocessing for text-to-speech
• Transliteration pipelines
• Code-switched Hindi/English text

Not intended for

• General language identification across arbitrary languages. Two labels only. Other Indian languages in Latin script will be forced into HIN or ENG.
• Devanagari transliteration itself. This model decides what to transliterate. It does not transliterate.
• Determining whether a whole sentence is Hindi or English. It is a token-level model. Aggregating its labels to a sentence verdict is not what it was built or evaluated for.

Limitations

SMS and chat shorthand. The largest out-of-domain error class is abbreviated social-media spelling, absent from tutoring text: ur, u, h, k, b, r, g. On chat-register text expect worse than the LinCE number suggests — that number already contains these errors, but diluted by well-formed tokens.

Short fragments. Accuracy tracks available context, not language. On short homograph-dense English — "Let me know so I can do it in the morning" — error rates rise sharply, because let, me, know, so, can, do are all Hindi words in other contexts and six words condition almost nothing. On full-length English text the false-Hindi rate is ~1%; on hand-picked short fragments it was 12.8%. Length is the variable.

Named entities are out of scope. The model emits only HIN/ENG and was never trained on an NE class. Personal and place names receive some label, and which one is not meaningful. Handle names separately.

Domain. Indian K-12 / exam-prep tutoring text: explanatory, second-person, mathematics- and science-heavy, generally well-formed sentences. That is where 0.9920 holds.

Data

The training data is proprietary tutoring text and is not released. No training data is included in this repository — weights, config and tokenizer only.

The data reflects the register, subject matter and code-mixing conventions of one setting: Hindi-English as used in Indian exam preparation. It is not a sample of Hinglish in general, and the model's notion of "which words are Hindi" is that community's.

Worth stating plainly, since this model exists to serve a TTS front-end: a HIN label causes a word to be transliterated to Devanagari and pronounced with Hindi phonology. A mislabelled word is therefore mispronounced, not dropped. The failure is audible — the right direction for a failure to go, but it does mean errors reach listeners directly.

License

Apache-2.0, inherited from google/muril-base-cased. Training data is not included and is not licensed for redistribution.

Citation

@misc{muril-hinglish-lid,
  title  = {muril-hinglish-lid: context-aware Hindi/English language identification for romanized Hinglish},
  author = {PhysicsWallah AI},
  year   = {2026},
  url    = {https://huggingface.co/PhysicsWallahAI/muril-hinglish-lid}
}

Base model:

@article{khanuja2021muril,
  title   = {MuRIL: Multilingual Representations for Indian Languages},
  author  = {Khanuja, Simran and Bansal, Diksha and Mehtani, Sarvesh and others},
  journal = {arXiv preprint arXiv:2103.10730},
  year    = {2021}
}