t5lemmatizer

Transformer based sanskrit lemmatizer used for obtaining bag of lemmas for BM25 retriever component of RAG. Use only one sentence at a time. Always end the input string with </s>.

Example usage:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("mahesh27/t5lemmatizer")
model = AutoModelForSeq2SeqLM.from_pretrained("mahesh27/t5lemmatizer", device_map="auto")

text = "sabhūmiṃviśvato'vṛtvātyatiṣṭhaddaśāṅgulam</s>"

inputs = tokenizer(text, return_tensors='pt')['input_ids'].to(model.device)

model.eval()
preds = model.generate(input_ids=inputs, max_new_tokens=48)
tokenizer.decode(preds, skip_special_tokens=True)[0]

'sa bhūmi viśvatas a vṛ ati sthā daśan aṅgula'

Citation:

@inproceedings{akavarapu-etal-2025-case,
    title = "A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in {LLM}s",
    author = "Akavarapu, V.S.D.S.Mahesh  and
      Terdalkar, Hrishikesh  and
      Bhattacharyya, Pramit  and
      Agarwal, Shubhangi  and
      Deulgaonkar, Dr. Vishakha  and
      Dangarikar, Chaitali  and
      Manna, Pralay  and
      Bhattacharya, Arnab",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.141/",
    doi = "10.18653/v1/2025.findings-acl.141",
    pages = "2745--2761",
    ISBN = "979-8-89176-256-5",
    abstract = "Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across diverse tasks and languages. In this study, we focus on natural language understanding in three classical languages{---}Sanskrit, Ancient Greek and Latin{---}to investigate the factors affecting cross-lingual zero-shot generalization. First, we explore named entity recognition and machine translation into English. While LLMs perform equal to or better than fine-tuned baselines on out-of-domain data, smaller models often struggle, especially with niche or abstract entity types. In addition, we concentrate on Sanskrit by presenting a factoid question{--}answering (QA) dataset and show that incorporating context via retrieval-augmented generation approach significantly boosts performance. In contrast, we observe pronounced performance drops for smaller LLMs across these QA tasks. These results suggest model scale as an important factor influencing cross-lingual generalization. Assuming that models used such as GPT-4o and Llama-3.1 are not instruction fine-tuned on classical languages, our findings provide insights into how LLMs may generalize on these languages and their consequent utility in classical studies."
}