mimic-vit-biogpt

mimic-vit-biogpt: a documented negative result

A ViT-encoder + BioGPT-decoder checkpoint for chest X-ray impressions whose decoder very likely ignores the image. Kept public as a reproducible record of a silent failure mode, not as a usable model.

Do not use this checkpoint for any image-conditioned task. It has not been evaluated and is not clinically validated.

TL;DR

• Built with VisionEncoderDecoder.from_encoder_decoder_pretrained(ViT, BioGPT) and add_cross_attention=True.
• The BioGPT implementation in Transformers has no cross-attention layers, and the flag is silently ignored (no error or warning). Image features are passed to the decoder but not used.
• The final model is therefore expected to behave like a text-only BioGPT fine-tuned for 500 steps on report impressions.
• A corrected pipeline with a held-out evaluation and a shuffled-image control is planned. Nothing here is a benchmark result.

Model details

What went wrong, and the evidence

The loss curve is weak evidence on its own, since a short run with a small learning rate could look similar for other reasons. The structural evidence is the strong part.

Reproduce the check:

import torch
from transformers import VisionEncoderDecoderModel

m = VisionEncoderDecoderModel.from_pretrained("Vikhram-S/mimic-vit-biogpt")

# 1) Does the decoder define any cross-attention parameters?
print([n for n, _ in m.decoder.named_parameters()
       if "cross" in n.lower() or "encoder_attn" in n.lower()])

# 2) Does the loss reach the image encoder?
m.train()
out = m(pixel_values=torch.randn(1, 3, 224, 224),
        labels=torch.randint(5, 1000, (1, 16)))
out.loss.backward()
n_grad = sum(p.grad is not None and p.grad.abs().sum() > 0
             for p in m.encoder.parameters())
print("encoder tensors with nonzero gradient:", int(n_grad),
      "of", len(list(m.encoder.parameters())))

# 3) Do different images give identical greedy output?
m.eval()
with torch.no_grad():
    a = m.generate(pixel_values=torch.randn(1, 3, 224, 224),
                   max_length=30, num_beams=1, do_sample=False)
    b = m.generate(pixel_values=torch.randn(1, 3, 224, 224),
                   max_length=30, num_beams=1, do_sample=False)
print("identical:", a.shape == b.shape and bool((a == b).all()))

Results of this script on the released weights will be added in a card revision.

Training details

The validation and test splits were not used, and no validation loss was computed.

Evaluation

None. No language-generation or clinical-label metrics were computed for this checkpoint. The 2,000-example test split of the source dataset is unused and is reserved for the corrected model.

Uses

• Appropriate: studying this failure mode, or as a minimal example of a silent cross-attention omission.
• Inappropriate: report generation, clinical or diagnostic use, or any claim about image-conditioned performance.

Not implemented

No explainability or attribution method, no confidence estimation, no retrieval grounding, no structured findings/impression output (the target was impressions only), and no alignment or consistency losses.

Bias, risks and limitations

• The decoder likely ignores the image, so outputs may be unrelated to the input.
• Very short training on a single-institution dataset processed by a third party; demographic and preprocessing bias are uncharacterized.
• No error analysis or hallucination analysis was performed.

Data and access terms

This repository contains weights only. MIMIC-CXR is distributed through PhysioNet under a credentialed-access agreement; users are responsible for complying with its terms, including when using processed copies.

Recommendations for people building similar models

1. Before training, assert that cross-attention parameters exist in the decoder.
2. After a few steps, check that the encoder receives nonzero gradients.
3. Always evaluate with a shuffled-image control; a model that scores the same with the wrong images is not using images.

Plan

A corrected model (a decoder with cross-attention, or an image-prefix design) trained and evaluated on the held-out test split, with the shuffled-image control, will be released as a separate repository. Results will be reported only with the script that produced them.

Citation

@misc{vikhram2026mimicvitbiogpt,
  author = {Vikhram S},
  title  = {mimic-vit-biogpt: a documented negative result},
  year   = {2026},
  howpublished = {Hugging Face model repository},
  url    = {https://huggingface.co/Vikhram-S/mimic-vit-biogpt}
}

Author

Vikhram S: Google Scholar | GitHub