mistral-medical-qlora

License

apache-2.0

Base_model

mistralai/Mistral-7B-v0.1

Tags medical,qlora,lora,peft,mistral

Datasets

medmcqa

Language

en

Mistral-7B Medical QLoRA

Fine-tuned version of Mistral-7B-v0.1 on the MedMCQA dataset using QLoRA for parameter-efficient domain adaptation to medical multiple-choice question answering.

Results

Model Accuracy (300-question held-out eval) Base Mistral-7B 41.0% Fine-tuned (this model) 52.7% Improvement +11.7% Training Details Base model: mistralai/Mistral-7B-v0.1 Method: QLoRA (4-bit NF4 quantization + LoRA adapters) LoRA config: rank=32, alpha=64, target modules = q_proj, k_proj, v_proj, o_proj Dataset: MedMCQA (8,000 training samples, 500 validation samples) Epochs: 1 Hardware: Single NVIDIA T4 GPU Training time: ~5 hours 18 minutes Final training loss: 1.026 Final validation loss: 1.130 Mean token accuracy: 75.8%

prompt = """You are a medical expert. Answer the following multiple choice question.

Question: A 45-year-old presents with chest pain radiating to the left arm. A) Angina B) Myocardial Infarction C) GERD D) Anxiety

Answer:"""

Intended Use

This model is intended for research and educational purposes — demonstrating parameter-efficient fine-tuning techniques on medical multiple-choice QA. It is not intended for clinical or diagnostic use and should not be used to make real-world medical decisions.

Limitations

Trained on only 8,000 of the 194,000 available MedMCQA samples due to compute constraints Single epoch of training; further epochs or full-dataset training would likely improve performance Evaluated using greedy decoding on the first generated token only, which may slightly undercount correct answers Not validated against clinical safety standards — outputs should not be used for actual medical advice

Links

GitHub repo: mistral-medical-qlora Full report: see GitHub repo for training curves and detailed methodology

Citation

If you reference this work, please cite:

@misc{usmani2026medicalqlora, author = {Asad Usmani}, title = {Domain Adaptation of Mistral-7B for Medical Question Answering using QLoRA}, year = {2026}, publisher = {HuggingFace}, howpublished = {\url{https://huggingface.co/Asad854/mistral-medical-qlora}} }