magi-onnx

magi — ONNX export (detection only, no OCR)

ONNX export of ragavsachdeva/magi, containing only the detection and association sub-networks. The OCR sub-network has been intentionally omitted so users can substitute their own OCR model appropriate to their context.

Files

Inference order

1. Preprocess the manga page image using MagiProcessor from the original repo (or reimplement: resize to 1024×1024, normalise, build pixel mask)
2. Run detection_transformer.onnx → last_hidden_state, reference_points
3. Slice object tokens: last_hidden_state[:, :-5] (last 5 are special tokens)
4. Run bbox_predictor.onnx and class_labels_classifier.onnx on object tokens
5. Crop detected character regions and run crop_embedding_model.onnx
6. Run character_character_matching_head.onnx to cluster characters
7. Run text_character_matching_head.onnx to associate speech bubbles to speakers
8. Run is_this_text_a_dialogue.onnx to filter non-dialogue text
9. Run your own OCR on the detected text bounding boxes

Source

Conversion script: github.com/masoniis/magi-onnx

License

The provided model and datasets are available for unrestricted use in personal, research, non-commercial, and not-for-profit endeavors. For any other usage scenarios, kindly contact me via email, providing a detailed description of your requirements, to establish a tailored licensing arrangement. My contact information can be found on my website: ragavsachdeva.github.io

Citation

@misc{sachdeva2024manga,
      title={The Manga Whisperer: Automatically Generating Transcriptions for Comics},
      author={Ragav Sachdeva and Andrew Zisserman},
      year={2024},
      eprint={2401.10224},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}