MedQwen-1.5B-LoRA-medrag

MedQwen-1.5B-LoRA-medrag

LoRA fine-tuned Qwen2.5-1.5B-Instruct on ~43K Chinese medical Q&A pairs derived from the MedGraphRAG knowledge base, with rank-16 adapters and an 80/10/10 train/val/test split.

Trained on a richer dataset than mellee030/MedQwen-1.5B-LoRA-r16 — 43K vs 24K pairs, sourced from 8,807 disease records across 14 question templates, filtered to answers ≥ 15 characters.

Evaluation Results

BERTScore uses bert-base-chinese. ROUGE uses character-level tokenization for Chinese. Evaluated on 200 randomly sampled held-out test pairs (never seen during training).

Dataset Comparison

Training Configuration

Dataset

54,095 Chinese medical Q&A pairs generated from the MedGraphRAG knowledge base (medical.json, 8,807 disease records). Each disease produces up to 14 Q&A pairs covering symptoms, causes, prevention, treatment, medication, diet, complications, and more. Pairs with answers shorter than 15 characters are filtered out to remove low-information entries. Split 80/10/10 with seed=42.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id    = "mellee030/MedQwen-1.5B-LoRA-medrag"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id, dtype=torch.float16, device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

messages = [
    {"role": "system", "content": "你是一个专业的医疗问答助手，请根据用户的问题给出准确、简洁的医疗建议。"},
    {"role": "user",   "content": "糖尿病的早期症状有哪些？"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Evaluation Methodology

Evaluation is run on a 200-sample random subset of the held-out test set (5,410 pairs, seed=42). The test set is never seen during training or early stopping — only used for final reporting.

For each sample:

1. The question is fed to the model with the system prompt via the Qwen2.5 chat template
2. The model generates a response greedily (do_sample=False, repetition_penalty=1.1, max_new_tokens=100)
3. The generated response is compared against the ground-truth answer using:
   • ROUGE-1/2/L — character-level n-gram overlap (Chinese has no word boundaries, so character tokenization is used)
   • BERTScore — semantic similarity via bert-base-chinese embeddings; F1 between contextualized token embeddings of prediction and reference

BERTScore is the primary metric because it captures semantic equivalence rather than surface overlap — a concise correct answer scores well even if it uses different phrasing than the reference.

Early stopping during training uses validation cross-entropy loss (not BERTScore), evaluated on the full 5,409-sample validation set every 1,000 optimizer steps.

Disclaimer

This model is for research purposes only and should not be used as a substitute for professional medical advice.