gemma-2b-it-marathi-lora

gemma-2b-it-marathi-lora

This repository contains LoRA PEFT adapter weights for google/gemma-2b-it fine-tuned on a Cleaned Alpaca Marathi-translated dataset.

This model attempts to reproduce the experimental setup described in: Khade et al., CHiPSAL 2025.

Model description

• Base model: google/gemma-2b-it
• Tuning Method: LoRA (Low-Rank Adaptation)
• Target Modules: q_proj, k_proj, v_proj, o_proj
• LoRA Hyperparameters: r = 16, alpha = 32, dropout = 0.05
• Language: Marathi (mr)
• Training data: Marathi-translated Alpaca instruction dataset (source)

Training Details

• Dataset: 51,760 instruction-response pairs translated into Marathi from the Cleaned Stanford Alpaca dataset(unsloth/alpaca-cleaned).
• Precision: Pure FP16 / BF16 (unquantized base weights).
• Epochs: 3
• Optimizer: AdamW

Intended use

Research and educational use, not a production-ready assistant.

Automated Evaluation Results

Evaluated using the AI4Bharat Airavata benchmark harness:

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/gemma-2b-it"
adapter_id = "lubzo/gemma-2b-it-marathi-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)

prompt = "खालील प्रश्नाचे उत्तर द्या:\nभारताची राजधानी कोणती आहे?\nउत्तर:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

## Limitations

- Trained on machine-translated data, not natively-authored Marathi — inherits translation artifacts and biases.
  

## Citation

```bibtex
@inproceedings{khade2025challenges,
  title={Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning},
  author={Khade, Omkar and Jagdale, Shruti and Phaltankar, Abhishek and Takalikar, Gauri and Joshi, Raviraj},
  booktitle={Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025)},
  year={2025}
}