biomedclip-cxr-lora

BiomedCLIP-CXR LoRA

LoRA adapter for a chest X-ray multi-label classifier built on BiomedCLIP's vision tower.

Only the vision tower is used — this is an image-only multi-label classifier, not a CLIP similarity model.

Files

Important: not a drop-in PEFT adapter

This adapter is part of a hand-reconstructed model graph, not an auto-loadable PEFT module. PeftModel.from_pretrained(...) will not work out of the box: the LoRA lives inside the fused qkv projection of a timm-wrapped ViT (visual.trunk), which has no stock PEFT target. You must rebuild the exact architecture before loading weights:

1. Load the base model via open_clip.create_model_and_transforms('microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224').
2. Freeze the vision tower (clip_model.visual).
3. Inject LoRA into Q and V only of the last 6 blocks (blocks[6..11]). timm backend: wrap blocks[idx].attn.qkv with a module that adds low-rank adapters to the Q and V slices of the fused qkv output (K slice unchanged). LoRA math: output = frozen_linear(x) + (alpha/r) * dropout(x) @ A^T @ B^T.
4. Append the classification head: nn.Sequential(LayerNorm(512), Dropout(0.1), Linear(512, 13)) over the encoder output (head input here is 512 — the output of encoder.head.proj).
5. load_state_dict the tensors in adapter_model.safetensors onto the reconstructed state dict.

Preprocessing

• Resize to 224×224, convert to RGB.
• Normalize with (0.4815, 0.4578, 0.4082) / (0.2686, 0.2613, 0.2758) (BiomedCLIP's own preprocessing stats).
• No augmentation at inference. model.eval(), torch.no_grad().

Output / labels

13 raw logits, one per class, in this exact order:

['enlarged cardiomediastinum', 'cardiomegaly', 'atelectasis', 'consolidation',
 'lung edema', 'fracture', 'lung lesion', 'pleural effusion', 'pneumonia',
 'pneumothorax', 'support device', 'lung opacity', 'pleural other']

Multi-label: apply a per-class sigmoid (no softmax).

For binary predictions, threshold each class's sigmoid probability using calibrated_thresholds.json (per-class, not a flat 0.5). If you only need a score, report raw sigmoid outputs.