MedQA-Llama3.1-8B-SFT-Big

MedQA-Llama3.1-8B-SFT-Big

QLoRA fine-tune of Llama-3.1-8B-Instruct on 50,000 trilingual medical Q&A (EN, FR, Moroccan Darija) augmented with the Dorosz Causal Knowledge Graph. Part of the BRAIN HEALTH / Operation HELIX-FT project.

Output format: model wraps the final answer in <answer>...</answer> and replies in the same language as the question (EN, FR, or Darija).

---

Training data

Training recipe (QLoRA)

Training metrics

The eval loss reached its minimum around step 5860 (epoch 1.96) at 0.7685. Beyond epoch 2.0 the model showed gradual overfitting (eval_loss ↑ to 0.82 at epoch 2.20). The load_best_model_at_end=True callback ensures the published adapter is the best checkpoint, not the final one.

Training curves

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(base, "BrainHealthAI/MedQA-Llama3.1-8B-SFT-Big")
tok = AutoTokenizer.from_pretrained("BrainHealthAI/MedQA-Llama3.1-8B-SFT-Big")

SYSTEM_FR = (
    "Vous êtes un assistant médical rigoureux. Répondez TOUJOURS en français. "
    "Raisonnez d'abord entre <think>...</think>, puis donnez la réponse finale "
    "entre <answer>...</answer>."
)

msgs = [
    {"role": "system", "content": SYSTEM_FR},
    {"role": "user",   "content": "Question : Quels sont les symptômes du diabète de type 2 ?"},
]
inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

For Darija (Arabic-script) questions, the model replies in Arabic-script Darija. For English questions, in English. The system prompt MUST instruct the language explicitly to avoid drift.

Companion model

For comparison, see Williamsanderson/MedQA-Llama3.1-8B-SFT-Small — trained on a different (smaller, single-language) corpus without KG augmentation.

Limitations

• Prototype R&D only — not a certified medical device. Always defer to a qualified physician.
• Possible hallucinations despite fine-tuning. Confidence calibration not yet evaluated.
• Darija outputs use Arabic script. Quality is uneven across the long tail of rare specialties (only ~877 trilingual samples / specialty).
• Training stopped at 2.20 epochs (early stop). A full 3-epoch run might marginally improve eval performance.

References

• QLoRA: Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. arXiv:2305.14314
• LoRA: Hu et al. (2022). arXiv:2106.09685
• Llama-3.1: Grattafiori et al. (2024). arXiv:2407.21783
• BRAIN HEALTH / Operation HELIX-FT — internal project document, MVP 2026.

Citation

@misc{medqa_sft_big_2026,
  title  = { MedQA-Llama3.1-8B-SFT-Big: Trilingual medical QA via QLoRA SFT on Llama-3.1-8B with Dorosz KG },
  author = { BRAIN HEALTH project — Operation HELIX-FT },
  year   = { 2026 },
  url    = { https://huggingface.co/BrainHealthAI/MedQA-Llama3.1-8B-SFT-Big }
}