medical-qa-lora

Medical QA LoRA

Model Description

This project fine-tunes Qwen2.5-1.5B-Instruct on medical educational text using LoRA (Low-Rank Adaptation) and supervised fine-tuning (SFT).

The goal is to adapt the language model to answer questions based on medical educational material while training only a small percentage of the model parameters.

Base Model

• Qwen/Qwen2.5-1.5B-Instruct

Fine-Tuning Method

• PEFT
• LoRA (Low-Rank Adaptation)
• Supervised Fine-Tuning (SFT)
• 4-bit quantization

Dataset

Medical educational reference material was processed and formatted into text samples for fine-tuning.

Training samples: 49
Validation samples: 6

Training

The model was fine-tuned using Hugging Face Transformers, TRL and PEFT.

LoRA was used to reduce the number of trainable parameters and memory requirements.

Evaluation Results

• Training Loss: 2.1675
• Validation Loss: 2.0194
• Mean Token Accuracy: 0.5213
• Evaluation Entropy: 2.0662

Intended Use

This model is intended for educational experimentation with LLM fine-tuning and medical question answering.

It should not be used as a substitute for professional medical advice, diagnosis or treatment.

Technologies

Python, PyTorch, Hugging Face Transformers, PEFT, LoRA, TRL, BitsAndBytes and Google Colab.