NeuroQformer-PubMed

NeuroQformer-PubMed

PubMed-grounded QFormer checkpoint from docs/02_models/13_qformer_pubmed.ipynb.

This is the checkpoint before anatomical/canonical fine-tuning. It is packaged with the image InfoNCE projection head, but canonical projection is disabled.

from neurovlm import NeuroVLM

nvlm = NeuroVLM(device="cuda")
text = nvlm.brain(img).to_text(
    method="generative",
    qformer_variant="pubmed",
    basis=None,
    prefix_text="",
)

Repo ID: neurovlm/NeuroQformer-PubMed

Revision 2026-10

Weights of the revised NeuroVLM paper. What changed:

• Deterministic training. Every stage runs with seeded data loaders and deterministic kernels. Two runs from scratch gave byte-identical weights on the same GPU model, driver and library versions.
• MLP projector. The brain-to-text projectors (stages 13 and 14) are residual MLPs (28.3 M parameters) instead of Q-Formers. A package names its projector in config.json["projector"].
• Stage-14 recipe. Each training text is projected onto the canonical bank of its own level, every training map is re-binarized per batch, and title tokens weigh 5 in the loss. Five seeds were trained and the one with the lowest validation loss (seed 3) is packaged.
• Calibration. Text-to-brain maps are multiplied by one global factor per model, fitted on the validation split and clipped at 1 (map_calibration.json, head 19.16 and adapter 6.13).
• Title decoding. After a [NETWORK] or [REGION] tag the decoder writes one of the canonical names in title_vocabulary.json (48 network and 83 region titles), so it never invents a title.

model.safetensors and config.json hold the stage-13 MLP projector (best validation epoch), with canonical projection off. Loading needs neurovlm with ResidualMLPProjector (October 2026, NeuroQFormer.from_state_dict_payload reads config.json["projector"]). load_projector in NeuroVLM's scripts/models/_common_models.py reads the package too.

Provenance: scripts/outputs/runs/final of NeuroVLM (scripts/pipeline/train_pipeline.py, finished 2026-10-10T19:56:25), training code at git 51bedf4 with 131 modified files, inputs pinned by scripts/config/hf_manifest.json (sha256 49f3e3a9825b). scripts/README.md records the later runs from scratch that gave the same bytes.

Earlier weights: revision 2493bafd8f41b9130d8d76832870c18bdcd8ef4c.