CheX-Phi35V

CheX-Phi3.5V — Preference-Optimised Vision-Language Model for Chest X-ray Interpretation

CheX-Phi3.5V is a vision–language model (VLM) that answers clinical questions about chest radiographs and generates structured findings reports.
Built on Phi-3.5 Vision-Instruct (7 B), it introduces Direct Preference Optimization (DPO) and contrastive rejection to achieve fine-grained medical reasoning while suppressing hallucinations.

---

Key Features

---

Quick Start

from transformers import AutoModelForVision2Seq, AutoProcessor

model_id  = "CheX-Phi-3.5-vision-instruct-DPO"
model     = AutoModelForVision2Seq.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

inputs = processor(remove4a
    images="example_frontal.jpg",
    text="Question: What abnormalities are present?\nAnswer:",
    return_tensors="pt"
).to(model.device)

generated = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(generated[0], skip_special_tokens=True))

Dependencies pip install transformers>=4.41.0 timm accelerate For batch inference or a Streamlit demo, see the scripts in the GitHub repo.

---

Available Checkpoints

---

Training Data & Procedure

Hardware : 8 × A100 80 GB • FP16 • DeepSpeed ZeRO-3 Total steps ≈ 2.2 M.

---

Evaluation

Evaluation scripts are provided in stage3_evaluate_mimic_ext_vqa.sh.

---

Ethical & Safety Considerations

• Clinical usage — Outputs are assistive only; a certified radiologist must confirm findings.
• Bias — Training data skewed towards North-American adult populations; paediatric or non-western cohorts may underperform.
• Privacy — MIMIC-CXR is fully de-identified; the model does not memorise PHI.
• Hallucinations — Contrastive rejection reduces but does not eliminate false positives; use confidence thresholds.

---

Known Limitations

1. No generalisation to CT, MRI, or ultrasound modalities.
2. Sensitive to extreme noise & portable AP projections.
3. Knowledge cutoff = Mar 2023; newly described conditions may be missed.

---

Resources

• Code & training scripts — https://huggingface.co/HaiZhiYan/CheX-Phi35V
• Data utilities — tools/generate_visual_prompt.py
• Demo notebook — demo.py

---

Citation

@misc{liu2025chexphi35v,
  title        = {CheX-Phi3.5V: Preference-Optimised Vision-Language Model for Chest X-ray Interpretation},
  author       = {Liu, Xiao and Others},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/HaiZhiYan/CheX-Phi35V/tree/main/CheX-Phi-3.5-vision-instruct-DPO}}
}

If you use CheX-Phi3.5V, please cite us and consider sharing your downstream results!