srt-cxr14-pooled-probe

Reading three frozen backbones together on ChestX-ray14

Three general-purpose multimodal models, none trained on radiology and none fine-tuned here, encode all 112,120 images of ChestX-ray14. One linear probe per backbone. Their logits are then averaged, which adds no parameters at all.

Official test_list.txt, 86,524 train / 25,596 test.

Split-matched baseline 0.7451: Wang et al. 2017 (arXiv:1705.02315v5, Table 17), a ResNet-50 fine-tuned end to end, on the same official test list.

The gain is significant: paired patient-clustered bootstrap on the difference of mean AUROCs gives +0.0124, 95% CI [+0.0082, +0.0168], positive in 1,000 of 1,000 resamples. Averaging logits adds no parameters, so the gain cannot be extra capacity. It is information one backbone holds and the others do not.

The clearest evidence for that: Aria scores 0.7080 on its own, which is 0.0371 BEHIND the baseline, and pooling it in still helps.

Concatenation loses, and the control says why

Concatenating features scored 0.7627. Concatenating the best backbone with itself scored 0.7626, at identical parameter count and with zero new information. The two differ by 0.0001, so the entire concatenation effect is width rather than content. It was never retuned, so read it as untuned rather than refuted.

The probes are interchangeable across backbones

A probe fitted on one backbone and read on another's states, through a ridge map fitted on training rows only:

Transport cost is -0.0071. It is negative: moving a probe between backbones costs nothing on average. Four of six cross directions beat the target backbone's own probe.

Usage

The checkpoint holds one weight matrix, bias, and the train-split mean and standard deviation per backbone. The normalisation ships with the weights because each backbone is standardised with its own training statistics, without which the pooled score is not reproducible.

import torch
ck = torch.load("pooled_probe.pt", map_location="cpu", weights_only=False)
logits = []
for tag, p in ck["probes"].items():
    x = (states[tag] - p["mu"]) / p["sd"]        # that backbone's raw states
    logits.append(x @ p["weight"] + p["bias"])
score = torch.stack(logits).mean(0)              # 3 x 14 -> 14

ck["findings"] gives the column order.

Scope, and what this is not

Detection, not early detection. These labels describe what is visible in the image in front of you. Nothing here speaks to catching disease before it is apparent, which needs longitudinal data with outcomes.

Not a diagnostic device. Research artifact. No clinical validation, no prospective evaluation, no regulatory clearance.

Labels are NLP-mined from radiology reports by the dataset authors. Every model on this benchmark inherits that ceiling.

Patient overlap between train and test is zero, asserted by the script, which refuses to run otherwise. Confidence intervals resample patients rather than images, because the unit that repeats is the patient.

A fourth backbone is missing. Mistral Small 3.1 was encoded but one shard wrote a truncated file and the chain deleted the good shards alongside it. That is our bug, now fixed, and the result here is three backbones rather than four.

Reproducing

scripts/cxr_probe_ensemble.py in the SRT repository, with the controls above built in. Sibling surfaces: the single-backbone gemma-4 probe is at RiverRider/srt-cxr14-linear-probe, and its states at RiverRider/srt-cxr14-frozen-probe.