medrag-llama33-70b-qlora

MedRAG — Llama 3.3 70B QLoRA Adapters for MedQA-USMLE

LoRA adapters (rank=16) fine-tuned on MedQA-USMLE with QLoRA. Achieves 78.48% accuracy on the MedQA-US test set — within 3 points of GPT-4 (83.97%). Combined with a FAISS RAG pipeline over medical textbooks, reaches 80.99%.

Full code and RAG pipeline: github.com/MartinLiarte/medqa-llm

How to Get Started

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "martinarra31/medrag-llama33-70b-qlora")
model.eval()

Training Details

Training Data

MedQA-USMLE-4-options — 9,160 train / 1,018 val / 1,273 test (4-option USMLE format).

Training Hyperparameters

Training Infrastructure

• Hardware: 4× NVIDIA L40S 48GB
• Multi-GPU: DeepSpeed ZeRO-2
• Training time: 4h 18min

Evaluation

Evaluated on the MedQA-US test set (1,273 questions, greedy decoding).

• Fine-tuned only: 78.48%
• Fine-tuned + RAG (FAISS over 18 medical textbooks): 80.99% Reference: Xiong et al., Benchmarking Retrieval-Augmented Generation for Medicine, ACL 2024. arxiv:2402.13178

Bias, Risks, and Limitations

These adapters are intended for research purposes only. The model should not be used for real clinical decision-making. Performance was evaluated on USMLE-style multiple choice questions and may not generalize to other medical tasks or real patient scenarios.

Model Card Authors

Martín Liarte — CS student at UPV (Valencia), specializing in Computing & AI.

Framework versions

• PEFT 0.13.0
• TRL 1.7.0
• Transformers 4.46.0