iSight-slide

iSight-slide: image-level assessment of immunohistochemistry staining

The image-level model of iSight, a system for automated assessment of immunohistochemistry (IHC) images and protein staining patterns. This repository carries the trained checkpoint and the training and inference code.

🔗 Links

• Code repository: github.com/zhihuanglab/iSight
• Training dataset: nirschl-lab/hpa10m
• Companion models: zhihuanglab/iSight-cell (per-cell staining), zhihuanglab/iSight-target (target-cell selection)

🎯 Prediction tasks

The model predicts five attributes of an IHC image at once:

The checkpoint also carries two auxiliary heads used as additional supervision while training — tissue type (58 classes) and malignancy (2 classes) — which code/scripts/inference.py reports alongside the three staining tasks.

🧠 Model

CLIP ViT-L/14-336 patch encoder over all 336 px tissue patches of an image. Every patch contributes all 576 of its ViT tokens; a gated attention module scores each token position and softmaxes across patches at that position, so pooling is per token rather than per patch. The pooled representation is the mean over tokens. Two conditioning signals are added to it: a text (context) branch encoding the query (tissue, diagnosis, gene), applied with dropout during training and off at inference, and a 39-way cell-type embedding. Linear heads produce the outputs above. Model version v3_all_tokens.

📦 Files

config.json                               model configuration (read this first; see Downloading)
checkpoints/iSight-slide.pth   trained weights (model state dict only)
code/
  model/patch_encoder_with_clam.py        encoder, all-token gated attention, conditioning, heads
  dataset/hpadataset.py                   HPA10M MIL dataset, tissue-mask patching
  train.py                                training (DDP, resumable)
  config/config.ini                       the configuration the checkpoint was trained under
  scripts/inference.py                    image-level inference
  tissue.py                               tissue mask
requirements.txt

⬇️ Downloading

from huggingface_hub import snapshot_download
local = snapshot_download("zhihuanglab/iSight-slide")        # config.json + checkpoint + code

or a single file:

from huggingface_hub import hf_hub_download
cfg  = hf_hub_download("zhihuanglab/iSight-slide", "config.json")
ckpt = hf_hub_download("zhihuanglab/iSight-slide", "checkpoints/iSight-slide.pth")

Please fetch config.json alongside the checkpoint: it carries the model settings the code reads, and it is the file the Hub uses to count downloads of this repository.

🚀 Running

pip install -r requirements.txt
export ISIGHT_DATA_ROOT=/path/to/hpa10m          # metadata, RLE masks, images
cd code
python train.py --config config/config.ini

Data locations are environment variables, not hard-coded paths:

The released configuration uses batch_size = 1, which is what the checkpoint was trained with.

📄 License

PENN Academic Software License Agreement: non-commercial research use only.

📧 Contact

Zhi Huang — [email protected]