clt-medgemma-cxr

Cross-Layer Transcoder for MedGemma (chest X-ray)

The first image-token-aware windowed cross-layer transcoder (CLT) trained on MedGemma activations over paired chest-X-ray and report data. Trained as part of TraceCXR, a mechanistic-interpretability investigation into report-prior hallucination — a vision-language model asserting findings its image does not support.

What it is

A sparse dictionary that reconstructs a layer's activations from a sparse feature basis, reading across a window of layers rather than a single one. Unlike a per-layer SAE, each feature can write into several downstream layers, which is what makes cross-layer circuit tracing possible.

Weights (params):

Reconstruction quality

Fraction of variance unexplained (FVU) on held-out activations — lower is better:

L0 (mean active features per token) ≈ 30. Reconstruction MSE ≈ 6e-4.

Image activations reconstruct less well than text, which is itself informative: the image stream carries more variance that a sparse basis of this width does not capture.

Loading

import torch
ckpt = torch.load("clt_medgemma_cxr.pt", map_location="cpu")
W_enc, b_enc = ckpt["params"]["W_enc"], ckpt["params"]["b_enc"]   # (12, 2560, 8192), (12, 8192)
W_dec, b_dec = ckpt["params"]["W_dec"], ckpt["params"]["b_dec"]   # (12, 3, 8192, 2560), (12, 2560)

# per layer L: features = relu(acts_L @ W_enc[L] + b_enc[L])
# the decoder then writes into layers L .. L+2 via W_dec[L]

Optimizer state has been stripped (the original checkpoint was 12GB; this is 4GB of weights only).

Honest limitations

• Reconstruction quality is not interpretability. A low FVU says the dictionary spans the activations, not that its features are monosemantic or causally meaningful. No feature-level interpretability audit is published with this checkpoint.
• The TraceCXR finding this was built to test — that report-prior hallucination is distributed rather than a clampable circuit — is a negative result. This transcoder is released as a reusable artifact, not as evidence that the intervention worked.
• Trained on one base model and one domain. Transfer to other VLMs or modalities is untested.

Citation

Part of the TraceCXR project (Kevin Jin, Duke University, 2026).