qwen3-8b-medmcqa-qlora

Qwen3-8B QLoRA — MedMCQA (research artifact)

⚠️ Not for clinical use

This adapter is not a medical device and must not be used for medical advice, diagnosis, or treatment. It has never been validated on real patients, real clinical notes, or any prospective data. It was trained on multiple-choice exam questions and is evaluated only on multiple-choice exam questions.

It is confidently wrong a great deal of the time. In the benchmark below, 86% of the base model's errors were high-confidence errors — the model does not hedge when it is mistaken. Any clinical deployment would require validation this artifact does not have and was never intended to support.

It exists to answer a research question — when is fine-tuning worth it versus just retrieving? — and its value is the comparison, not the checkpoint.

Model summary

A LoRA adapter for Qwen/Qwen3-8B, trained with QLoRA (4-bit NF4 base) on MedMCQA training questions. Released as one arm of a controlled six-arm benchmark, not as a general medical assistant.

Intended use

In scope. Reproducing the benchmark in the accompanying repository; research on fine-tuning versus retrieval trade-offs; a baseline for further medical-QA experiments.

Out of scope. Any clinical, diagnostic, triage, or treatment context. Patient-facing applications. Generating medical content for consumption without expert review. Any use where being confidently wrong carries a cost.

Results

Scored on a frozen, subject-stratified 1,000-item test set carved from MedMCQA's labelled validation split. (MedMCQA's own test split has cop == -1 — labels are withheld — so it cannot be used for evaluation.)

+6.1 points over base, paired McNemar p < 0.0001 over identical items.

The headline finding is not flattering to this adapter. Retrieving the same training explanations at inference time scores higher (67.0%) than absorbing them into these weights (62.9%), and the gap holds across three training seeds (+4.2 points, 95% CI [+1.0, +7.2], p = 0.008). Same base model, same prompt, same information — one in an index, one in the weights, and the index wins.

The adapter's advantage is cost, not accuracy: merged, it has identical inference latency to the base model while serving 113-token prompts instead of 656. It repays its training cost at roughly 200,000 queries.

Training data

MedMCQA, Apache 2.0 — Indian medical entrance exam questions (AIIMS/NEET-PG). No PHI: exam questions with de-identified vignettes.

Preprocessing, all documented and reproducible:

• OCR repair. MedMCQA's text has systematic extractor corruption in which the rt bigram is dropped, and the corrupted spelling is more common than the correct one (aery 11,203 vs artery 7,009; hea 7,626 vs heart 4,957). Repaired with a 225-entry human-reviewed lexicon covering 93,174 tokens.
• Answer-key stripping. Many explanations open with the answer letter ("Ans. is 'd' i.e., ..."). Training on those teaches the letter, not the medicine, so the boilerplate is removed and explanations reduced to nothing are dropped.
• Leakage gate. Rows are filtered on content hash, not id — MedMCQA repeats items across splits under different ids. 0 leaked rows reached training.
• Subsampling. A seeded random 30,000 of the ~179,600 clean rows.

Targets are the answer letter followed by the explanation. The letter comes first so that first-token scoring grades this arm identically to every other arm; the explanation is included so the fine-tune sees the same text the retrieval arm serves.

Limitations

1. Contamination is present, and measured. MedMCQA predates Qwen3 and is a widely mirrored public benchmark, so it almost certainly appears in pretraining. Three probes were run rather than assumed:

   • Verbatim reproduction. ~9% of test stems are regenerated well above a shuffled-reference chance baseline (base 10.0% vs 1.3% control; this adapter 8.7% vs 2.0%). Some absolute accuracy above is recall.
   • Option permutation. Shuffling answer positions moves the base model by -1.6 points — it is not relying on memorised answer labels. This adapter loses 3.2 points, so fine-tuning introduced a mild positional shortcut the base model did not have.
   • Position bias. Negligible for both (max excess 0.029 and 0.045).

   Because contamination is roughly constant across arms — they share a base model and a test set — the between-arm comparisons are largely unaffected, which is what the benchmark exists to measure. Absolute accuracy is inflated; the deltas are not. Full detail in REPORT.md.

2. Three training seeds, reported at one. Every table reports seed 42. Retrained at seeds 1 and 2, this recipe scores 62.9%, 63.9% and 61.7% (62.8 ± 1.1). Differences between configurations smaller than that spread — including every LoRA-rank and epoch ablation in the repository — are not evidence of anything.

3. The MCQ gain does not survive free-text generation. Every accuracy above is 4-option multiple choice scored by constrained log-probability. Asked the same 300 test questions open-ended (options hidden) and graded by a cross-family 70B judge against the gold answer, this adapter scores below the base model it was trained from — 0.396 vs 0.454 graded (p = 0.024), −4.0 points and not significant when scored strictly as correct-or-not — while retrieval over the same explanations gains +9.3 graded / +12.0 strict. A 50-item human check of the judge gave κ = 0.51 on the three-way rubric and 0.76 on correct-or-not; the judge is lenient on partial credit, which is why both scorings are reported. The adapter learned to rank candidates and an exam-answer register, not the facts needed to produce an answer. A stricter replication on MIRIAD (the same passages in the weights and in the index) puts the same adapter design significantly below base under both scorings (−8.8 graded, −7.3 strict). Do not use this adapter for generation; it is an MCQ artefact. No citation or calibration assessment beyond confidence margins. Full detail in REPORT.md §2.4–2.5.

4. 31.3% of the evaluation pool is dentistry, not general medicine. Results excluding it are meaningfully different (base 59.8%, this adapter 66.5%).

5. English only, and the questions reflect the Indian medical curriculum.

6. Merged evaluation. Latency figures assume the adapter is merged into the base. Served unmerged it is ~4.9× slower.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-8B"
REV = "b968826d9c46dd6066d109eabc6255188de91218"  # pragma: allowlist secret

tok = AutoTokenizer.from_pretrained(BASE, revision=REV)
model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype="bfloat16")
model = PeftModel.from_pretrained(model, "<this-repo>").merge_and_unload()

The benchmark pins enable_thinking=False in the chat template for every arm, because variable-length reasoning traces confound latency and cost measurement. Reproducing the numbers above requires the same setting.

Licences

• This adapter: Apache 2.0, inherited from the Qwen3-8B base.
• Training data: MedMCQA, Apache 2.0.
• Benchmark code: MIT.

MedQA was deliberately not used anywhere in this project: bigbio/med_qa declares its licence "unknown", and a re-uploader's cc-by-4.0 tag does not launder upstream copyright on USMLE board-prep material.

Citation

@software{koli_finetune_vs_rag_2026,
  author = {Koli, Viresh},
  title  = {Fine-Tune vs. Retrieve: A Controlled Benchmark in Clinical QA},
  year   = {2026},
  url    = {https://github.com/vireshkoli/Fine-Tune-vs-RAG}
}