phi3-medical-qlora

Phi-3 Mini Medical QLoRA

Model Overview

This model is a domain-adapted version of Microsoft's Phi-3 Mini 4K Instruct model, fine-tuned on medical question-answer pairs using QLoRA (Quantized Low-Rank Adaptation).

The goal of this project is to improve the model's ability to generate accurate and contextually relevant medical responses while maintaining the efficiency of parameter-efficient fine-tuning.

Base Model

• Base Model: microsoft/Phi-3-mini-4k-instruct
• Architecture: Phi-3 Mini (3.8B parameters)
• Context Length: 4K tokens
• Fine-Tuning Method: QLoRA
• Task: Medical Question Answering

Dataset

Training was performed using:

• Dataset: medalpaca/medical_meadow_medical_flashcards
• Domain: Medical Education and Healthcare
• Training Samples: ~10,000 examples
• Evaluation Samples: Held-out validation split

Data Format

Each example was formatted as:

Instruction:

<medical question>

Input:

<context if available>

Response:

<ground truth answer>

Training Configuration

Quantization

• 4-bit NF4 Quantization
• Double Quantization Enabled
• Compute Type: FP16

LoRA Configuration

• Rank (r): 32
• Alpha: 64
• Dropout: 0.05
• Bias: None

Target Modules

• q_proj
• k_proj
• v_proj
• o_proj
• gate_up_proj
• down_proj

Training Hyperparameters

• Epochs: 2
• Learning Rate: 1e-4
• Batch Size: 4
• Gradient Accumulation Steps: 4
• Max Sequence Length: 512
• Warmup Ratio: 0.03
• Optimizer: AdamW
• Evaluation Strategy: Step-based Validation

Evaluation Results

Evaluation was performed on a held-out validation set.

Perplexity

Improvement:

• 53.8% reduction in perplexity

ROUGE-L

Improvement:

• 76.7% increase in ROUGE-L

BERTScore

Key Findings

• Significant reduction in perplexity on medical validation samples.
• Improved overlap with reference medical answers measured by ROUGE-L.
• Better adaptation to healthcare-specific terminology and concepts.
• Achieved strong performance gains using parameter-efficient fine-tuning without modifying the full 3.8B model.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel

BASE_MODEL = "microsoft/Phi-3-mini-4k-instruct"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained( BASE_MODEL )

model = PeftModel.from_pretrained( base_model, "YOUR_USERNAME/phi3-medical-qlora" )

prompt = """

Instruction:

What are the symptoms of diabetes?

Input:

Response:

"""

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate( **inputs, max_new_tokens=150 )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• This model is intended for educational and research purposes.
• Outputs should not be used as professional medical advice.
• The model may generate incorrect or outdated medical information.
• Human verification by qualified healthcare professionals is recommended.

Disclaimer

This model is not a medical device and should not be used for diagnosis, treatment, or clinical decision-making.

Author

Javvadi Naga Venkata Anjaneya Prasad

Acknowledgements

• Microsoft Phi-3 Team
• Hugging Face
• PEFT
• TRL
• Medical Meadow Dataset Contributors