pcare-dim4-phase2-comparator

PCare DIM4 Phase 2 — Clinical Outcome Comparator

Fine-tuned Qwen3-8B with LoRA (r=256, alpha=256, all-linear) for binary clinical outcome comparison (PASS/FAIL).

Task

Compare an expected clinical outcome (from document analysis) against an actual clinical outcome (from a compiled FHIR artifact) and judge semantic equivalence.

Input: Two clinical outcome strings
Output: JSON with verdict (PASS/FAIL), reasoning, and mismatches

Evaluation Results

All 4 primary targets met. The model achieves perfect accuracy with zero safety misses on the 299-example eval set spanning 8 clinical domains and 15 scenario types.

Confusion Matrix (Fine-tuned)

Training Loss Trajectory

Training Details

• Base model: Qwen/Qwen3-8B (Apache 2.0)
• Method: SFT with LoRA (r=256, alpha=256, all-linear targets, dropout=0.05)
• Dataset: chrischarts8/pcare-dim4-phase2-synthetic — 1,196 train / 299 eval examples
• Epochs: 3
• Effective batch size: 32 (4 × 4 grad accum × 2 GPUs)
• Learning rate: 2e-4 (cosine schedule, 10% warmup)
• Loss: NLL with assistant_only_loss=True
• Hardware: 2×A10G (48GB total)
• Trainable parameters: 698M / 8.9B total (7.86%)
• Training time: ~1.5 hours

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "chrischarts8/pcare-dim4-phase2-comparator")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

system_prompt = """You are a clinical outcome comparator. Compare two sets of clinical outcomes for the same patient. Outcomes may be recommendations, diagnostic classifications, risk levels, or other clinical decisions.

## Rules
1. Different clinical outcomes = FAIL (even if both are 'reasonable').
2. Same clinical outcome with different wording = PASS.
3. One outcome is more specific than the other but clinically consistent = PASS.
4. Missing an outcome domain that the expected has = FAIL.
5. Extra outcome domains not in the expected are acceptable = PASS.
6. If the expected says 'not indicated' / 'excluded' and actual says the equivalent = PASS.

## Output Format
Return ONLY a JSON object:
{"verdict": "PASS" or "FAIL", "reasoning": "brief explanation", "mismatches": ["list of mismatches, empty if PASS"]}"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "## Expected Outcome (from document analysis)\nScreen for colorectal cancer with colonoscopy every 10 years, starting at age 45\n\n## Actual Outcome (from compiled artifact)\n[Colorectal cancer screening] Colonoscopy every 10 years — ages 45-75 (Grade A)"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"verdict": "PASS", "reasoning": "Both outcomes recommend colorectal cancer screening with colonoscopy every 10 years starting at age 45. The actual is more specific (adds age range and Grade A) but clinically consistent.", "mismatches": []}

Clinical Domains Covered

1. Cancer screening — colorectal, breast, cervical, lung, prostate
2. Diagnostic classification — transfusion reactions (TRALI/TACO), sepsis (qSOFA/SOFA), stroke (TOAST)
3. Treatment guidelines — antibiotics, anticoagulation, pain management
4. Risk stratification — cardiovascular (Framingham, ASCVD), fall risk, nutrition
5. Immunization schedules — childhood, adult, travel vaccines
6. Preventive care — wellness visit components, lifestyle counseling
7. Referral criteria — specialist referral thresholds
8. Monitoring protocols — lab monitoring frequency, imaging follow-up

Comparison Rules Learned