Llama-3.2-3B-Instruct-Odia

Llama-3.2-3B-Instruct-Odia

This model is a fine-tuned version of Meta's Llama-3.2-3B-Instruct optimized for Odia language generation and understanding. It was trained using the Unsloth library to efficiently fine-tune the model with LoRA on the Alpaca-Odia dataset.

Model Description

• Base Model: Meta's Llama-3.2-3B-Instruct
• Training Technique: QLoRA fine-tuning with Unsloth
• Training Dataset: Alpaca-Odia with data augmentation
• Language Capabilities: Odia (ଓଡ଼ିଆ) and English

This model has been specifically optimized to:

• Understand instructions in both English and Odia
• Generate fluent responses in Odia
• Respond to various queries with culturally appropriate Odia text

Intended Uses

• Odia language generation and conversation
• Question answering in Odia

Training Details

The model was fine-tuned using:

• Parameter-efficient fine-tuning with QLoRA (r=32)
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 2e-4 with linear scheduler
• Training with 4-bit quantization for memory efficiency
• Data augmentation to improve instruction following in both languages

Sample Usage

from transformers import pipeline
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "sumankumarbhadra/Llama-3.2-3B-Instruct-Odia"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto",
    low_cpu_mem_usage=True
)

pipe = pipeline(
    "text-generation", 
    model=model,
    tokenizer=tokenizer
)

messages = [
    {"role": "user", "content": "Hi, How are you?"},
]

result = pipe(
    messages, 
    do_sample=True,
    max_length=512,
    num_return_sequences=1,
    batch_size=1
)

assistant_response = result[0]['generated_text'][-1]['content']
print(assistant_response)

Limitations

• The model's Odia language capabilities are limited by the size and quality of the training dataset
• Performance varies based on the complexity and domain of the query
• May occasionally mix English and Odia in responses for certain prompts

Ethical Considerations

This model aims to increase accessibility to AI technologies for Odia speakers. When using this model, please ensure:

• Respect for cultural sensitivities and local contexts
• Awareness of potential biases in generated content
• Proper content moderation for deployed applications
• Acknowledgment of limitations when using for critical applications

Acknowledgements

• Meta for releasing the Llama 3.2 model family
• Unsloth team for their efficient fine-tuning library