Downloads · 30 days
3
3% of all-time downloads
michelleAnogueira/biomistral-medquad-lora
biomistral-medquad-lora is a text generation model from michelleAnogueira. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
This is a fine-tuned version of BioMistral-7B on the MedQuAD dataset (16,325 medical Q&A pairs) using QLoRA (4-bit quantization + LoRA adapters).
Downloads · 30 days
3
3% of all-time downloads
All-time downloads
96
Public
Repo size
168 MB
Likes
0
Public
Click a slice to open those files.
.safetensors168 MB · 98%
From the Hugging Face model README
This is a fine-tuned version of BioMistral-7B on the MedQuAD dataset (16,325 medical Q&A pairs) using QLoRA (4-bit quantization + LoRA adapters).
The model was fine-tuned as part of the Tech Challenge FIAP - Phase 3 (Final project for "AI for Developers" course) to build a medical assistant for Brazilian Portuguese healthcare professionals.
This model is designed to answer medical questions in English in a structured, clinical format inspired by NIH MedQuAD. It can be used for:
This model is part of a larger medical assistant pipeline that includes:
⚠️ This model is NOT a substitute for medical professionals. It is an academic prototype and should NEVER be used for:
Users (both direct and downstream) must:
pip install torch transformers peft bitsandbytes accelerate
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"BioMistral/BioMistral-7B",
load_in_4bit=True, # QLoRA: 4-bit quantization
device_map="auto",
)
# Load LoRA adapter (THIS REPO)
model = PeftModel.from_pretrained(base_model, "michelleAnogueira/biomistral-medquad-lora")
tokenizer = AutoTokenizer.from_pretrained("michelleAnogueira/biomistral-medquad-lora")
# Alpaca-style prompt template
prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
What are the symptoms of diabetes?
### Response:
"""
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.5,
top_p=0.9,
do_sample=True,
repetition_penalty=1.3,
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response.split("### Response:")[-1].strip())
### Instruction:, ### Response:)| Hyperparameter | Value | Justification |
|---|---|---|
| Method | QLoRA (4-bit) + Unsloth | Fits in 40GB A100 GPU |
| LoRA rank (r) | 16 | Balance between capacity and overfitting |
| LoRA alpha | 32 | Convention: alpha = 2 × rank |
| LoRA dropout | 0.05 | Light regularization |
| Target modules | q, k, v, o, gate, up, down | All linear layers |
| Learning rate | 2e-4 | Standard for LoRA (QLoRA paper) |
| LR scheduler | cosine | Smooth decay |
| Warmup steps | 50 | Stable training start |
| Optimizer | adamw_8bit | Memory-efficient |
| Epochs | 2 | Sweet spot for 14k samples |
| Batch size | 2 (per device) | VRAM limit |
| Gradient accumulation | 4 | Effective batch = 8 |
| Max sequence length | 4096 | Fits longest MedQuAD samples |
| Metric | Value | Interpretation |
|---|---|---|
| Training Loss (final) | ~0.50 | Model adapted to format |
| Validation Loss | 0.5864 | Good generalization |
| Validation Perplexity | 2.18 | Excellent (hesitates between ~2 words) |
| Base Model Perplexity (val) | 4.31 | Baseline reference |
| Perplexity Reduction | 49.4% | Fine-tuning cut hesitation in half |
| Train-Val Gap | 1.47× | Small gap, no harmful overfitting |
15 out-of-distribution tests performed:
Conclusion: No harmful overfitting. The model maintains general medical knowledge while adopting the MedQuAD response style.
@misc{biomistral-medquad-2026,
author = {Michelle Almeida Nogueira Rodrigues},
title = {BioMistral-7B Fine-Tuned on MedQuAD for Medical Question Answering},
year = {2026},
publisher = {HuggingFace},
howpublished = {\\url{https://huggingface.co/michelleAnogueira/biomistral-medquad-lora}},
note = {Tech Challenge FIAP - Phase 3}
}
Nogueira Rodrigues, M. A. (2026). BioMistral-7B Fine-Tuned on MedQuAD for Medical Question Answering. HuggingFace. https://huggingface.co/michelleAnogueira/biomistral-medquad-lora
Michelle Almeida Nogueira Rodrigues
[email protected] (via GitHub: @MichelleANogueira)
Disclaimer: This is an academic project. The model is provided "as-is" without warranty. Not for production medical use without extensive validation, regulatory approval, and ongoing human oversight.