paLLM_adalimumab_RA_Lora

Introduction

Prior authorization (PA) is a crucial but labor-intensive process in pharmacy benefit management (PBM) that ensures appropriate use of specialty and high-cost medications. For drugs like Adalimumab (Humira), PA review requires expert pharmacists to assess documentation against strict criteria—such as disease severity, step therapy history, and lab screening results—resulting in workflow bottlenecks and potential delays in patient care. Large language models (LLMs), while powerful, often struggle with domain-specific tasks like PA due to their lack of fine-tuning on structured clinical policies and decision logic. In this project, I developed a domain-adapted LLM (paLLM) to support PA decision-making for Adalimumab in rheumatoid arthritis.

Data

To fine-tune my model on prior authorization (PA) decisions for Adalimumab in rheumatoid arthritis (RA), I generated a custom dataset of 1,000 synthetic chart note summaries with associated PA decisions and rationales. Each example includes a concise patient summary (with diagnosis, treatment history, TB/HBV screening, dosing, etc.) and a structured response with a binary decision, rationale, and evidence quotes from the chart. Synthetic examples were created using Claude 3 Sonnet, prompted with 2-shot examples (one approved, one denied) and randomized section phrasing to prevent templating. The final dataset was evenly split between approvals and denials, with diverse edge cases. I performed an 80/10/10 split into training (800), validation (100), and test (100) sets, using a fixed random seed for reproducibility. To validate generation quality, I used GPT-5 as an LLM-as-a-judge, calibrated on a gold set of 50 pharmacist-written cases. The judge re-adjudicated each synthetic case and scored faithfulness and sufficiency; flagged items were audited manually. All data files used are included in the data directory of this repository, in both .jsonl and .csv format.<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Methodology

Parameter-efficient fine-tuning using LoRA to teach the policy directly to the small model without relying on large prompt contexts.

Training Hyperparameters

• Trial 1: r=8, alpha=16, dropout=0.05
• Trial 2: r=16, alpha=32, dropout=0.05
• Trial 3: r=32, alpha=32, dropout=0.1
• For all the trials: learning rate = 0.00001, eval_steps = 10, epochs = 2

Selected Hyperparameters

• Among three LoRA configurations tested, the best setup (Trial 1) used r=8, alpha=16, and dropout=0.05, achieving the lowest training and validation loss over two epochs.
• Trial  1 achieved the lowest training loss (2.66) and validation loss (2.69) with stable convergence.
• Trials  2 and  3 produced slightly higher losses and showed no improvement despite larger parameter sizes.
• This configuration balanced model capacity with overfitting risk on my small, balanced dataset. Training inputs consisted of chart note summaries paired with sectioned PA criteria, and outputs included binary decisions and short rationales.

Evaluation: Metrics

• Label metrics: accuracy, precision, F1 score
• LLM-as-a-judge scores: ensuring that the model's predictions were not only correct, but also well-grounded in policy and documentation.
  • Faithfulness (0 or 1): Does the rationale align with evidence and policy?
  • Sufficiency (0 or 1): Is the explanation complete enough to support the decision?
  • Clarity (1–5): Is the output clearly written and understandable?
• External Benchmark
  • MMLU: General-domain reasoning
  • MEDMCQA: Medical multiple-choice accuracy
  • RACE: Reading comprehension

Results

Model Selection

I tested training-free prompting (zero, 3 shot, 8 shot)on three instruct models, Llama-3.2-1B, Qwen-2.5-7B, and Mistral-7B. While the 7B models produced the most accurate approve/deny calls, Llama-3.2-1B consistently returned clean JSON aligned with schema but misapplied the policy (signs of overfitting to 8 shots wordings). I think this is a kind of deficiency LoRA can fix with a small, targeted update. Llama-3.2-1B is also fast, cheap, and easy to run on hardware, which helps with latency and PHI needs(patient health information). So I will keep the small model and teach it the policy with LoRA instead of relying on larger prompts or larger models.

Metric Results

Summary

• Performance on label metrics and custom metrics: After training LoRA on Llama-3.2-1B-Instruct, we see clear improvements in how the model applied the policy on the testing split. Before fine-tuning, the model returned clean JSON but was biased toward approving all cases, resulting in 100% recall but 0% precision for the “deny” class, and faithfulness and sufficiency scores were quite low (0.17 and 0.02). After LoRA fine-tuning using a 50:50 balanced dataset and strict output formatting, the model produced more stable outputs, improved JSON faithfulness (0.40), sufficiency (0.30), and clarity (3.0), and increased the F1 score for “approve” from 0.68 to 0.75. While it still predicted only one class (approve), the LoRA model handled language alignment better, showing better rationales and structure. On the internal policy task, LoRA training yielded more faithful, sufficient, and interpretable outputs (faithfulness 0.17 → 0.40, sufficiency 0.02 → 0.30, clarity 2.42 → 3.00), with a higher F1 score for “approve” (0.68 → 0.75).
• Perforamance on external benchmarks: After LoRA fine‑tuning, the model maintained stable overall benchmark performance. MMLU 46.1% → 46.1%, MedMCQA 41.4% → 41.7%, and RACE 38.1% → 37.8%. it indicates that LoRA adaptation improved domain‑specific reasoning without harming general knowledge retention.
• Conclusion: Overall, LoRA helped close the policy gap while keeping the model fast, small for real-world pharmacy use.

Usage and Intended Uses

This model, baicalein/paLLM_adalimumab_RA_Lora, is a domain-adapted LoRA fine-tuned version of Llama-3.2-1B designed to assist with prior authorization (PA) review for Adalimumab (Humira) in rheumatoid arthritis. It accepts a clinical chart note and drug-specific PA criteria as input and returns a binary approval decision along with a brief rationale, structured in JSON format. The model is intended for use in research and workflow prototyping within managed care and specialty pharmacy settings. It is not approved for production clinical use or real patient data.

Ideal Use Cases

• Simulating pharmacist decision logic for clinical NLP tasks
• Evaluating model-generated chart notes summary
• Serving as a PA policy reasoning baseline in LLM benchmarks

How to Load

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-1B-Instruct", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-1B-Instruct")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "baicalein/paLLM_adalimumab_RA_Lora")

# Example inference
input_text = """You are a PA reviewer. Given the following chart note and criteria, return a JSON response:
[chart_note] ...
[pa_criteria] ..."""

inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Prompt Format

The model expects an input prompt that includes a chart note summary followed by drug‑specific PA criteria, and it must be formatted exactly as shown below. The model is trained to return a strict JSON object with fields for decision and rationale, so prompts should clearly separate the chart note and criteria sections.

You are a prior-authorization reviewer.  
Given the following **chart note** and **PA criteria**, return a JSON object  
with the fields: diagnosis, step_therapy, safety, dosing, decision, rationale.

[CHART_NOTE]
52-year-old female with seropositive RA. Prior MTX 15mg weekly × 14 weeks with inadequate response.
No prior biologics. TB test negative 2023-08-12. Hep B surface antigen negative 2023-08-10.
Plan: Start Adalimumab 40 mg every 2 weeks.

[PA_CRITERIA]
1. Diagnosis of moderate–severe RA  
2. Step therapy: MTX ≥12 weeks or contraindication  
3. TB and Hep B screening required  
4. Dosing: 40 mg SC every 2 weeks

Return your decision strictly based on evidence in the chart note.

Expected Output Format

The model returns a structured JSON object that summarizes the PA decision rationale. Each field maps to a specific coverage criterion, allowing downstream systems to parse and audit the model’s reasoning.

Approval Example

{
  "diagnosis": "RA with moderate–severe activity",
  "step_therapy": "Tried MTX ≥12 weeks with inadequate response",
  "safety": "TB and Hep B screening completed and negative",
  "dosing": "Adalimumab 40 mg SC every 2 weeks",
  "decision": "approve",
  "rationale": "All four PA criteria are clearly met. The patient has seropositive RA, completed MTX step therapy, passed TB and Hep B screening, and the dosing aligns with labeling."
}

Denial Example

{
  "diagnosis": "RA with moderate activity",
  "step_therapy": "MTX started only 4 weeks ago",
  "safety": "TB screening done; Hep B screening missing",
  "dosing": "Adalimumab 40 mg SC every 2 weeks",
  "decision": "deny",
  "rationale": "Step therapy is incomplete (MTX <12 weeks), and Hep B screening is missing. These must be completed before approval per criteria sections 2 and 3."
}

Limitations

• The small model size (Llama-3.2-1B) constrains its capacity to generalize to complex or ambiguous cases, especially those requiring nuanced clinical reasoning or interpretation of conflicting evidence.
• Despite LoRA fine-tuning, the model occasionally misinterprets policy sections when phrasing is altered or documentation is borderline incomplete. Benchmark metrics for faithfulness and sufficiency, while improved post-tuning, remain below optimal thresholds for clinical deployment.
• Output consistency depends on prompt structure and is sensitive to deviations from the trained format.
• External benchmark scores (MMLU, MedMCQA, RACE) did not improve, suggesting that domain-specific gains came at no cost—but also with no generalization benefit.
• This model was trained only on synthetic chart notes; future iterations should incorporate deidentified real-world PA cases for higher realism and robustness.
• the LLM-as-a-judge component relied on a single human pharmacist (myself) for calibration and spot auditing. Involving multiple practicing PA pharmacists would provide more robust and diverse evaluation.
• In addition, because PA criteria are subject to frequent updates across plans and payers, a retrieval-augmented generation (RAG) framework may offer better adaptability than static fine-tuning alone.

Model Card Authors

Angie H Yoon, Pharm D, BCPS, BCOP, CSP, MS

Model Card Contact

[email protected], [email protected]