AGPVT

Attention Gating PVT(AGPVT)

Simultaneous Segmentation and Classification Deep Learning Model for Esophageal Lesions

Supported disease:

Normal Esophagus(Health), Esophageal Protruded Lesions, Esophagitis, Barrett Esophagus, Esophageal Cancer

Use it with released weights:

  from transformers import AutoModel
  model = AutoModel.from_pretrained("gepeixuan/AGPVT", trust_remote_code=True)
  # see demo to get your input_tensor
  outputs = model(input_tensor)
  pred_segmentation_mask, pred_classification = outputs['seg'].softmax(dim=1),  outputs['cls'].softmax(dim=1) #in batch

Demo:

Check preprocessing setup and demos at https://huggingface.co/spaces/gepeixuan/AGPVT_demo

Recommend dataset for testing:

MICCAI 2015 Endoscopic Vision Challenge: https://endovissub-barrett.grand-challenge.org/

Endoscopy Disease Detection and Segmentation (EDD2020): https://edd2020.grand-challenge.org/

Other information:

The initial backbone weights can be downloaded from: https://github.com/whai362/PVT

Current version is V2, replacing the old CBAM attention with MHSA for better performance

Train dataset can only be requested for research use.

Please Cite

If you find our work helpful, please consider citing our paper:

@ARTICLE{10742420,
  author={Ge, Peixuan and Yan, Tao and Wong, Pak Kin and Li, Zheng and Chan, In Neng and Yu, Hon Ho and Chan, Chon In and Yao, Liang and Hu, Ying and Gao, Shan},
  journal={IEEE Transactions on Emerging Topics in Computational Intelligence}, 
  title={Simultaneous Segmentation and Classification of Esophageal Lesions Using Attention Gating Pyramid Vision Transformer}, 
  year={2024},
  volume={},
  number={},
  pages={1-15},
  keywords={Lesions; Image segmentation; Esophagus; Transformers; Cancer; Hospitals; Computer vision; Accuracy; Multitasking; Germanium; Medical image classification; medical image segmentation; multi-task learning; esophageal lesion; transformer},
  doi={10.1109/TETCI.2024.3485704}
}