IndicPhi-mini

IndicPhi-mini: Adapting Phi-mini-MoE to Indic Languages with Curated Data

Overview

IndicPhi-mini is a fine-tuned version of Microsoft’s Phi-mini-MoE, a compact Mixture-of-Experts (MoE) model, adapted specifically for Indic languages. It is trained on a curated multilingual dataset of approximately 29 million high-quality samples, standardized into a conversational format from diverse sources. By leveraging efficient fine-tuning techniques such as QLoRA-based quantization and LoRA adapters, the model enhances Indic language capabilities while keeping resource usage practical. Evaluation on benchmark datasets shows consistent 3–4% accuracy improvements across multiple Indic languages, demonstrating the effectiveness of targeted fine-tuning with curated data. a compact Mixture-of-Experts (MoE) model

---

Key Contributions

• Curated one of the largest Indic corpora to date: 561M samples → cleaned into 29M high-quality samples across 13 Indic languages.
• Fine-tuned Phi-mini-MoE (7.6B params, 2.4B active) using QLoRA (4-bit) and LoRA adapters, making training feasible on a single A100-80GB GPU.
• Achieved +3–4% accuracy improvements on major Indic benchmarks:
  • ARC-Challenge-Indic (reasoning tasks)
  • MMLU-Indic (knowledge & domain understanding)
• Improved generalization across multiple Indic languages including Hindi, Kannada, Tamil, Telugu, Marathi, Bengali, Malayalam, Gujarati, Odia, Punjabi, Assamese, Sinhala, and Urdu.

---

Model Architecture

• Base model: Phi-mini-MoE-Instruct (Microsoft)
• Parameters: 7.6B total (2.4B active per token)
• Layers: 32 decoder-only transformer blocks
• Attention: Grouped Query Attention (GQA)
• Experts per layer: 16 (Top-2 active per token)
• Context length: 4096 tokens

---

Usage

To load the fine-tuned model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "SandLogicTechnologies/IndicPhi-mini"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    load_in_4bit=True
)

prompt = "ग्रामीण क्षेत्रों में ऑनलाइन शिक्षा की समस्याएं क्या हैं?"  

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Dataset Preparation

Data Sources

• Total collected: 561M samples from 53 datasets from Hugging Face.
• Languages covered: 13 Indian languages which include Hindi, Kannada, Telugu, Tamil, Marathi, Malayalam, Gujarati, Bengali,Odia, Punjabi, Assamese, Sinhala, Urdu.
• Categories: General text, translation, instruction, conversational.

Processing Pipeline

1. Manual Filtering – removed noisy, irrelevant, and malformed samples.
2. Preprocessing – deduplication, language identification, normalization, minimum length filtering.
3. Format Conversion – standardized into UltraChat JSON schema (multi-turn conversations).

Final Cleaned Dataset

• Size: 29M samples

Dataset Distribution (Final Cleaned)

Total curated dataset: ~29 million high-quality samples

---

Training Details

• Hardware: 1 × NVIDIA A100-80GB
• Precision: QLoRA (4-bit quantization)
• Batching: Effective batch size 256 (32 × 8 gradient accumulation)
• Steps: 8,500
• Optimizer: AdamW (8-bit) + cosine LR schedule + 1k warmup steps
• LoRA configuration:
  • Layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  • r=128, α=128, dropout=0
• Final training loss: 0.48

---

Evaluation & Results

Benchmarks

1. ARC-Challenge-Indic (reasoning)
2. MMLU-Indic (knowledge & domain understanding)

Improvements

• ARC-Challenge-Indic
  • Accuracy: 21.03 → 24.46 (+3.43%)
  • Normalized Accuracy: 24.69 → 28.86 (+4.17%)
• MMLU-Indic
  • Accuracy: 27.47 → 30.95 (+3.48%)

Results

ARC-Challenge-Indic

Accuracy: (Phi-mini-MoE) 21.03 → (IndicPhi-mini) 24.46 (+3.43%)

MMLU-Indic

Accuracy: (Phi-mini-MoE) 27.47 → (IndicPhi-mini) 30.95 (+3.48%)

Acknowledgments

The Phi-mini-MoE-Instruct models are based on the original work by Microsoft and fine-tuned by the Sandlogic development team.

Special thanks to:

• The Microsoft team for developing and releasing the microsoft/Phi-mini-MoE-instruct model.
• The authors and organizations behind the 53 open-source datasets that made this work possible.
  The complete list of dataset sources and citations is available here.

---

Contact

For any inquiries or support, please contact us at [email protected] or visit our Website.