phi3-auditor-merged

🏥 phi3-auditor-merged

Phi-3-mini fine-tuned for clinical AI model auditing.

This model takes a JSON object of ML performance metrics (AUC, ECE, drift, label shift, etc.) and returns a structured health classification label plus a detailed explanation — helping teams audit deployed clinical models for drift, calibration failure, class imbalance, and other issues.

---

Model Details

---

Intended Use

What this model does

Given a JSON report of clinical ML model performance metrics, the model:

1. Assigns a Category label (e.g. Calibration Failure, Major Drift, Class Imbalance Problem, Healthy)
2. Generates a concise Explanation with observations and recommendations

Intended users

• ML engineers monitoring deployed clinical models
• Healthcare data science teams running periodic model audits
• Researchers studying automated model health assessment

Out-of-scope use

• Not suitable for direct clinical decision-making or patient diagnosis
• Not a replacement for domain expert review of model performance
• Not designed for non-clinical ML tasks
• Should not be used on data types outside its training distribution (non-tabular metrics, images, etc.)

---

How to Use

Requirements

pip install transformers torch accelerate

Basic inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "PhantomAjusshi/phi3-auditor-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,  # Required for custom Phi-3 modeling files
)

report = """{
  "auc": 0.863,
  "accuracy": 0.83,
  "precision": 0.79,
  "recall": 0.69,
  "f1": 0.79,
  "ece": 0.278,
  "brier": 0.263,
  "drift": 0.03,
  "missing_rate": 0.003,
  "label_shift": 0.06,
  "pos_rate": 0.10,
  "data_integrity_issues": 0
}"""

prompt = (
    f"<|system|>\nYou are a clinical AI auditor model.\n"
    f"<|user|>\nInstruction: Analyze the clinical model report and classify its health.\n\nReport:\n{report}\n"
    f"<|assistant|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=400,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.2,
        do_sample=True,
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
# Extract only the assistant's reply
reply = response.split("<|assistant|>")[-1].strip()
print(reply)

Expected output format

Category: Calibration Failure
Explanation: High calibration error (ECE 0.278) despite reasonable discrimination (AUC 0.863).
The model's probability outputs are poorly aligned with actual outcomes. Recommend
recalibration using Platt scaling or isotonic regression, and threshold review.

Input metrics reference

---

Training Details

Dataset

• Name: Custom synthetic clinical audit dataset (audit_dataset_v2_5000.json)
• Size: 5,000 labeled samples
• Split: 80% train (4,000) / 20% test (1,000)
• Format: JSONL — each record has instruction, input (metrics JSON), output (category + explanation)
• Generation date: November 17, 2025

Each sample pairs a set of synthetic model performance metrics with a human-written audit label and explanation covering categories such as:

• Healthy / Passing
• Calibration Failure
• Major Drift / Potential Drift
• Class Imbalance Problem
• Data Integrity Issue
• Needs Review / Critical Failure

Training procedure

The base model was loaded in 8-bit using BitsAndBytesConfig and adapted with LoRA targeting the attention projection layers (q_proj, k_proj, v_proj, o_proj). After training, the LoRA adapter was merged into the base model weights using peft.merge_and_unload() and saved as full FP16 safetensors.

Prompt format used during training:

<|system|>
You are an AI auditor analyzing clinical model performance reports.
<|user|>
Instruction: Analyze the clinical model report and classify its health.

Report:
{ ...metrics JSON... }
<|assistant|>
Category: <label>
Explanation: <explanation>

Hyperparameters

Training loss

Loss converged rapidly after the first 150 steps, stabilizing around 0.41 for the remainder of training.

---

Evaluation

The model was evaluated on a held-out test set of 1,000 samples using weighted precision, recall, F1, and accuracy computed by extracting the Category: field from generated outputs and comparing to ground-truth labels.

Formal evaluation metrics will be added here once a full benchmark run is completed.

---

Limitations & Bias

• Synthetic training data: The model was trained entirely on synthetically generated audit reports. Real-world clinical model metrics may follow different distributions or contain edge cases not represented in training.
• Label sensitivity: The model may be sensitive to metric combinations near decision boundaries between categories.
• No temporal reasoning: The model does not reason about metric trends over time — each inference is based on a single snapshot of metrics.
• English only: All training data is in English.
• Not a substitute for expert review: Outputs should be treated as decision-support, not a final audit verdict.

---

Repository & Related Work

• Training code: Hospital-Audit-Trained-Model (GitHub)
• Web application: Hospital-Model-Audit-Website (GitHub) — a full-stack Next.js + FastAPI interface that uses this model via llama.cpp

---

Citation

If you use this model in your work, please cite:

@misc{phi3-auditor-merged,
  author       = {PhantomAjusshi},
  title        = {phi3-auditor-merged: Phi-3-mini fine-tuned for clinical AI model auditing},
  year         = {2025},
  publisher    = {HuggingFace},
  url          = {https://huggingface.co/PhantomAjusshi/phi3-auditor-merged}
}

---

License

This model is released under the MIT License.

The base model (microsoft/Phi-3-mini-4k-instruct) is subject to Microsoft's Phi-3 license. Please review it before use in commercial or production settings.