MarkushGrapher-2

<p align="center"> <img src="markushgrapher_2_repo_banner.png" alt="MarkushGrapher-2 Banner" width="100%"> </p>

MarkushGrapher-2 is an end-to-end multimodal model for recognizing chemical structures from patent document images. It jointly encodes vision, text, and layout information to convert Markush structure images into machine-readable CXSMILES representations.

Overview

MarkushGrapher-2 is a transformer-based model that integrates two complementary encoders:

• A Vision Encoder (Swin-B ViT), pretrained for Optical Chemical Structure Recognition (OCSR) (taken from Model: MolScribe*)
• A Vision-Text-Layout (VTL) Encoder (T5-base), trained for Markush feature extraction

The model also includes ChemicalOCR, a dedicated OCR module fine-tuned for chemical images, enabling fully end-to-end processing without external OCR dependencies.

*MolScribe: https://github.com/thomas0809/MolScribe

Architecture

The input image is processed through two parallel pipelines:

1. Vision pipeline: The image is encoded by the OCSR vision encoder and projected via an MLP.
2. VTL pipeline: The image is passed through ChemicalOCR to extract text and bounding boxes, which are then fused with image patches in the VTL encoder.

The outputs of both pipelines are concatenated and fed to a text decoder that autoregressively generates a CXSMILES sequence describing the Markush backbone and a substituent table.

Two-Stage Training Strategy

• Phase 1 (Adaptation): The Vision Encoder is frozen while the projector and text decoder are trained on 243K real-world image-SMILES pairs for standard molecular structure recognition.
• Phase 2 (Fusion): The VTL encoder is introduced and trained jointly with the text decoder on 235K synthetic and 145K real-world Markush structure samples for CXSMILES and substituent table prediction.

Performance

Markush Structure Recognition (CXSMILES Accuracy)

Molecular Structure Recognition (SMILES Accuracy)

Model Details

• Parameters: 831M total (744M trainable)
• Vision Encoder: Swin-B ViT (from MolScribe, frozen during Phase 2)
• VTL Encoder/Decoder: T5-base architecture with UDOP fusion
• OCR Module: ChemicalOCR (256M parameters, based on SmolDocling)
• Input: Patent document image crop (1024x1024)
• Output: CXSMILES sequence + substituent table

Usage

# Load the model checkpoint
from transformers import AutoModel

model = AutoModel.from_pretrained("docling-project/MarkushGrapher-2")

For full inference pipeline usage, see the MarkushGrapher repository.

Datasets

Training and evaluation datasets are available at docling-project/MarkushGrapher-2-Datasets.

Citation

@inproceedings{strohmeyer2026markushgrapher2,
  title     = {MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures},
  author    = {Strohmeyer, Tim and Morin, Lucas and Meijer, Gerhard Ingmar and Weber, Valery and Nassar, Ahmed and Staar, Peter W. J.},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

License

This model is released under the Apache 2.0 License.