llama-3.2-3b-dolly-qa

Llama 3.2 3B — Dolly QA Fine-tune

Model Description

This is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct, trained on the Databricks Dolly 15k dataset using QLoRA.

The model has been fine-tuned for question answering tasks — given an instruction and optional context, it produces concise, grounded answers.

Training Details

• Base model: meta-llama/Llama-3.2-3B-Instruct
• Dataset: databricks/databricks-dolly-15k (2000 examples)
• Method: QLoRA (4-bit quantization + LoRA adapters)
• LoRA rank: 16, alpha: 32
• Target modules: q_proj, k_proj, v_proj, o_proj
• Trainable parameters: 9M / 3.2B (0.28%)
• Epochs: 1
• Hardware: Kaggle T4 GPU
• Final training loss: 1.78

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("Soumil23/llama-3.2-3b-dolly-qa")
tokenizer = AutoTokenizer.from_pretrained("Soumil23/llama-3.2-3b-dolly-qa")

Developer

Soumil — CS Engineering student at Manipal University Jaipur