Llama-3.2-3B-Instruct-Odia-lora

sumankumarbhadra/Llama-3.2-3B-Instruct-Odia-lora

This repository contains the LoRA adapters for fine-tuning Meta's Llama-3.2-3B-Instruct on Odia language generation. These adapters can be applied to the base model to enable Odia language capabilities without the full fine-tuned model.

Model Description

• Base Model: Meta's Llama-3.2-3B-Instruct
• Adapter Type: Low-Rank Adaptation (LoRA) weights
• Training Dataset: Alpaca-Odia with data augmentation
• Language Capabilities: Odia (ଓଡ଼ିଆ) and English
• LoRA Configuration:
  • Rank (r): 32
  • Alpha: 32
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  • Dropout: 0

Training Details

The model was fine-tuned using:

• Parameter-efficient fine-tuning with LoRA (r=32)
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 2e-4 with linear scheduler
• Training with 4-bit quantization for memory efficiency
• Data augmentation to improve instruction following in both languages

Limitations

• Requires access to the base Llama-3.2-3B-Instruct model
• Performance depends on proper integration with the base model
• May require additional configurations for optimal performance on specific hardware

Acknowledgements

• Meta for releasing the Llama 3.2 model family
• Unsloth team for their efficient fine-tuning library