tinyllama-medqa-qlora

tinyllama-medqa-qlora

A QLoRA adapter (4-bit NF4, double-quantised) for TinyLlama-1.1B-Chat, fine-tuned for one epoch on MedQuAD medical Q&A. It is the treatment arm of a controlled comparison against LoRA on a smaller model — the other arm is Babblu2821/gpt2-medqa-lora.

Code, method and full results: https://github.com/fayazhussain2821/llm-finetuning-medqa

⚠️ Do not use this for medical information

This is a methodology demonstration, not a medical tool. Its factual reliability has been measured, and while it is the best arm in this project, it is still poor: on a blinded review of 20 held-out questions it contradicted the reference answer or invented an entity in 35% of them, scoring 3.25 out of 5 for factual soundness.

It produces fluent, confident, well-formed text that is frequently wrong. Fluency is exactly what makes this dangerous.

Do not use it for diagnosis, treatment, triage, patient-facing text, or to answer any real health question.

What it is for

Reproducing and studying a parameter-efficient fine-tuning comparison. Note that the project's own conclusion is a caution against over-reading this adapter: most of its advantage over the LoRA arm comes from the base model being ~9× larger and already instruction-tuned, not from QLoRA. See "Evaluation".

Training

Provenance. These weights were trained on 2026-08-03 with the project's original notebook pipeline, using TRL's SFTTrainer. The current repository trains both arms under transformers.Trainer — a deliberate change, so the two arms of the comparison no longer run different training machinery. Re-running the current code will not reproduce these exact weights. Every published number below was measured on these files.

Evaluation

Scored on 1,641 held-out rows, identical rows for every arm, answer span only — the prompt template differs between arms and scoring it would let boilerplate move the metric.

Bits per byte is the headline metric, not perplexity. Perplexity is per token, and TinyLlama's SentencePiece tokenizer differs from GPT-2's byte-level BPE, so the two perplexities are not on one scale.

Fine-tuning cut bits per byte by 35.4% against its own base model.

Read the control row carefully. Untouched TinyLlama (0.6120) is already within a few percent of fully fine-tuned GPT-2 (0.5970), having seen no MedQuAD at all. Most of the cross-model gap is model scale and prior instruction tuning, not QLoRA. An earlier version of this project reported a "53% improvement" by comparing the two fine-tuned arms' perplexities directly; that number was wrong twice over — different tokenizers, and no control.

Generated-answer quality

Greedy decoding, ≤200 new tokens, 200 held-out questions:

No degeneration — the model does not loop.

Factual soundness (blinded, 1–5)

Fine-tuning produced no detectable improvement in factual accuracy (paired 95% CI −0.20 to +0.90, spanning zero), despite the automatic metrics reporting large gains. What is detectable is that the untouched base model already beats the fully fine-tuned GPT-2 arm by +1.10 (95% CI +0.35 to +1.85).

These ratings are a human pass, rated blind to which model produced each answer, by the repository's author — one non-expert rater, not a clinician and not adjudicated by a second. An earlier LLM-judge pass over the same sheet scored every arm lower (2.70 mean, 40% contradiction for this model) but reached every identical verdict; both are published in the source repository.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
base = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(base, "Babblu2821/tinyllama-medqa-qlora")
tokenizer = AutoTokenizer.from_pretrained("Babblu2821/tinyllama-medqa-qlora")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is anemia?"}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=160, repetition_penalty=1.15)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

4-bit loading requires CUDA (bitsandbytes). On CPU or Apple Silicon, load in fp16 or fp32 as above — the published evaluation was run that way, in fp16 on MPS, so the figures are not bit-identical to a 4-bit run.

Limitations

• Not factually reliable. See the measured numbers above.
• One epoch, one seed, one run — no variance estimate across training runs.
• The comparison against the GPT-2 arm confounds model size, prior instruction tuning and adaptation method. Only the within-model comparison is attributable.
• MedQuAD is NIH-sourced, US-centric, and frozen at collection time.
• Evaluation compares against a single reference answer.

License

Apache 2.0, following the base model. The dataset carries its own license.