IndicVedas-LoRA

🙏 VedaGPT LoRA Adapters (IndicVedas-LoRA)

This repository contains the LoRA (Low-Rank Adaptation) adapter weights for VedaGPT (IndicVedas). These adapters are trained on top of the base model Qwen2.5-14B-Instruct using the Unsloth library.

If you are looking for the fully merged model (including GGUF quantizations for Ollama/llama.cpp), please check shinigamiRaj/IndicVedas.

---

⚙️ LoRA Configuration

• Rank (r): 64
• Alpha (lora_alpha): 64
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• LoRA Dropout: 0 (Optimized for Unsloth)
• Bias: none

---

📖 Training Recipe

The adapters were fine-tuned using serverless cloud GPUs via Modal:

• Dataset: Comprehensive Vedic and Ayurvedic digital corpus (Rig Veda, Sama Veda, Yajur Veda, Atharva Veda, Charaka Samhita, Sushruta Samhita, Rasa Jala Nidhi, etc.).
• Epochs: 2.0 full epochs.
• Learning Rate: 5e-5 (Linear warm-up for 5% of training steps, followed by Cosine decay).
• Optimizer: adamw_8bit.
• Weight Decay: 0.05 for generalization stability.
• Effective Batch Size: 8.
• Context Length: 4096 tokens.

---

🛠️ How to load the LoRA Adapters

You can load these adapters in Python using transformers and peft, or directly using unsloth for accelerated performance.

Loading with Unsloth (Recommended)

from unsloth import FastLanguageModel

max_seq_length = 4096
dtype = None # Auto detect
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "shinigamiRaj/IndicVedas-LoRA",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

# Prepare for inference
FastLanguageModel.for_inference(model)

# Generate text
inputs = tokenizer(
    ["<|im_start|>system\nYou are VedaGPT...\n<|im_end|>\n<|im_start|>user\nTell me about Agni in Rig Veda.<|im_end|>\n<|im_start|>assistant\n"], 
    return_tensors = "pt"
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens = 512, use_cache = True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Loading with standard PEFT & Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "Qwen/Qwen2.5-14B-Instruct"
adapter_model_name = "shinigamiRaj/IndicVedas-LoRA"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_model_name)

---

📝 Prompt Format

This model uses the ChatML markup structure:

<|im_start|>system
You are VedaGPT, an expert scholar of the ancient Vedic scriptures...<|im_end|>
<|im_start|>user
Describe the concept of digestion (Agni) in Charaka Samhita.<|im_end|>
<|im_start|>assistant
...