smolvlm-256m-cxr-vqa

SmolVLM-256M Chest X-Ray VQA (LoRA, Dual-View)

A LoRA fine-tuned version of SmolVLM-256M-Instruct for visual question answering over chest radiographs, trained on frontal + lateral image pairs from the IU X-Ray dataset.

⚠️ Research and educational use only. This model was built as a learning project on a small (~3,800 study) dataset and has not undergone clinical validation. Do not use it for real diagnostic or clinical decision-making. See Limitations below.

Model Details

• Base model: SmolVLM-256M-Instruct (268M parameters total)
• Fine-tuning method: LoRA, rank 16, alpha 32, applied to 210 language-model linear layers (4.46% of parameters trainable); the vision encoder was kept frozen and the connector/projector was fully trainable
• Input: one or two chest X-ray images per question (frontal view, optionally + lateral view)
• License: Apache 2.0, inherited from the base model
• Trained by: Nikhil00781
• Hardware: single 8 GB consumer laptop GPU (NVIDIA RTX 5050), ~7.5 hours for 3 epochs

Uses

Direct Use

Answering natural-language questions about chest X-ray images: yes/no questions about specific findings (e.g., "Is there evidence of cardiomegaly?"), listing visible abnormalities, and open-ended findings/impression summaries — intended for research, experimentation, and educational demonstration of vision-language model fine-tuning.

Out-of-Scope Use

• Clinical diagnosis or any real patient care decision
• Use as a medical device or diagnostic aid without appropriate regulatory clearance
• Deployment on imaging protocols, patient populations, or equipment substantially different from the IU X-Ray training distribution
• Any use where an incorrect answer could cause harm

Bias, Risks, and Limitations

• Small training dataset: fine-tuned on IU X-Ray (~3,800 studies from a single institution), far smaller and less diverse than datasets like MIMIC-CXR used in most published chest X-ray AI work. Performance and generalization to other hospitals' imaging or reporting styles is untested.
• Decoding strategy matters a lot for this model. During evaluation, greedy decoding was found to cause the model to collapse onto a small number of generic answers for open-ended questions, substantially understating its actual capability and recall on clinically important conditions. Use sampling-based decoding (see recommended settings below), not greedy decoding, for open-ended generation.
• Frozen vision encoder: the LoRA configuration used does not adapt the vision encoder, which may limit how fully the model exploits the second (lateral) image relative to a fully fine-tuned model.
• Class imbalance: the training data is imbalanced toward abnormal studies (~64%) and toward a long tail of rare finding labels; performance on rare conditions is less reliable than on common ones (reflected in the gap between micro- and macro-averaged clinical metrics reported below).
• No fairness, demographic, or subgroup analysis has been performed.

Recommendations

For open-ended questions (findings/impression), generate with sampling rather than greedy decoding:

model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7, top_p=0.9)

How to Get Started with the Model

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import torch
 
model_id = "your-username/smolvlm-256m-cxr-vqa"  # update to your actual repo id
processor = AutoProcessor.from_pretrained(model_id)
processor.image_processor.do_image_splitting = False  # important: disable tiling
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16)
 
frontal = Image.open("frontal.png").convert("RGB")
lateral = Image.open("lateral.png").convert("RGB")
 
messages = [{
    "role": "user",
    "content": [
        {"type": "image"}, {"type": "image"},
        {"type": "text", "text": "What is the impression of this chest x-ray?"},
    ],
}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[[frontal, lateral]], return_tensors="pt")
 
output = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7, top_p=0.9)
answer = processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer[0])

Training Details

Training Data

Derived from IU X-Ray (OpenI), a public collection of chest radiograph studies paired with free-text radiology reports. Question-answer pairs were generated with a rule-based, label-grounded procedure rather than an LLM, to avoid introducing hallucinated ground truth: closed-form questions were generated from structured MeSH-derived finding labels, and open-ended questions used the report's own Findings/Impression text verbatim as the answer. A label-cleaning step removed 13 bare anatomical terms (e.g., "diaphragm," "lung") that had been miscategorized as abnormal findings in the source data. The training set contains 14,574 question-answer pairs from 3,062 studies; data was split by study (not by individual question) to prevent leakage between train/validation/test.

Training Procedure

Evaluation

Evaluated on a held-out test set of 382 studies (991 yes/no, 134 list, 717 open-ended questions) using sampling-based decoding (temperature 0.7, top-p 0.9) for both this model and the un-fine-tuned baseline, to ensure a fair comparison.

CheXbert-F1 and RadGraph-F1 are structured, clinically-grounded metrics (rather than plain word-overlap) that specifically check whether a finding's presence, absence, or uncertainty is stated correctly — used here because standard n-gram metrics were found to be blind to negation errors (e.g., scoring "cardiomegaly is present" as nearly identical to its reference "no cardiomegaly is present").

Citation

If you use this model, please cite the underlying base model and dataset:

@misc{smolvlm2024,
  title={SmolVLM: small yet mighty Vision Language Model},
  author={HuggingFaceTB},
  year={2024},
  howpublished={\url{https://huggingface.co/blog/smolvlm}}
}
 
@article{demner2016preparing,
  title={Preparing a collection of radiology examinations for distribution and retrieval},
  author={Demner-Fushman, Dina and Kohli, Marc D and Rosenman, Marc B and Shooshan, Sonya E and Rodriguez, Laritza and Antani, Sameer and Thoma, George R and McDonald, Clement J},
  journal={Journal of the American Medical Informatics Association},
  volume={23}, number={2}, pages={304--310}, year={2016}
}

Model Card Contact

See the full project writeup and code at [https://github.com/Nikhil00781/SmolVLM-VQA-Project].