qwen3-0.6b-clinical-lora

Clinical LoRA adapter for Qwen3-0.6B

RESEARCH USE ONLY. Not a medical device. Not validated for clinical use. Must not be used to inform patient care.

Open weights and training logs for a LoRA adapter trained on clinical question-answer data, released so the result reported in our paper can be audited against the artefacts that produced it.

What the comparison means

Both legs trained the same recipe on the same three cards, one at a time and each alone on the machine. Held-out loss differs by 0.0042, about 0.19%, both bests at step 3,500, both scoring the same capped 512-example prefix of the same held-out split. The purpose was not to win: it was to test whether two independent training stacks reach the same place on a realistic workload. They do.

logs/ carries the per-step record, the supervising gate's verdicts and the reference leg's own curve, so the claim is checkable rather than merely stated.

Training data

Three public corpora, converted into Qwen3's chat template with loss on the answer only:

168,473 training and 3,440 held-out examples. The held-out split is a fixed prefix, never trained on. MedQA-USMLE is CC-BY-4.0 and is attributed here accordingly.

Using it

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
model = PeftModel.from_pretrained(base, "cloudkites/qwen3-0.6b-clinical-lora")

Prompts must use Qwen3's own chat template, including the empty <think>\n\n</think> block the model opens its final assistant turn with. Training with the wrong template cost 5.8% of held-out loss in an earlier run of this workload, and a mismatch at inference will cost something similar.

Limits

A 0.6 B base with a 10 M-parameter adapter, trained for one epoch on multiple-choice and abstract-derived clinical QA. It answers in the style of its training data and will be confidently wrong outside it. It is not aligned, not calibrated, and carries whatever biases its corpora carry. Held-out cross-entropy is the only metric reported; no clinical accuracy, safety or fairness evaluation has been performed.

Training framework

Trained with numbat, a machine-learning stack written in Zig with no third-party runtime dependencies, driven entirely through its C ABI. This repository contains weights, logs and documentation only -- no framework source, and none is needed to use the adapter.

Licence

Adapter weights under the Apache License 2.0, matching the base model.

Citation

@misc{cloudkites_qwen3_clinical_lora,
  title  = {Clinical LoRA adapter for Qwen3-0.6B},
  author = {Tran, Thang and Dang, Lan},
  year   = {2026},
  note   = {CloudKites AI Lab. Research use only. Open weights, Apache-2.0.}
}