rfdetr-instance-segmentation-leibniz-manuscripts

Zone detection model for (17th–18th c.) manuscripts, trained on Leibniz's manuscripts using SegmOnto ontology.

This model was developed by Denisa-Florina BUMBA as part of the ERC Philiumm project (Grant ADG No. 101020985), at the Laboratoire SPHERE, Université Paris Cité – CNRS, France.

Fine-tuned RF-DETR-Seg (Nano) for instance segmentation and zone detection on scanned pages of Leibniz's manuscripts. The model detects zones such as main text, margin text, numbering, diagrams, digitization artifacts, and several classes of mathematical formulae using the SegmOnto controlled vocabulary, in order to support automatic transcription in the digital humanities. It is trained specifically on Leibniz's manuscripts, but may be applied to other historical manuscripts with similar layout.

Dataset

Available at : rfdetr-segmentation-leibniz-dataset

Classes

Class names follow the SegmOnto controlled vocabulary for manuscript layout annotation, which is widely used across the HTR/OCR and the digital humanities community (Kraken, eScriptorium, HTR-United).

Training Data

Preprocessing:

• Auto-orient applied
• Resized (stretch) to 728×728

Augmentation (applied to the training set only, 3 augmented outputs generated per source image):

• Horizontal and vertical flip
• Rotation: ±3°
• Shear: ±5° horizontal, ±5° vertical
• Hue: ±15°
• Brightness: ±15%
• Noise: up to 1.49% of pixels

Training Parameters

The published weights are from epoch 43, the best raw checkpoint (segm_mAP_50:95 = 0.415), and from the best EMA checkpoint, at epoch 49 (segm_mAP_50:95 = 0.410).

Performance Metrics

checkpoint_best_regular.pth (epoch 43) — Overall metrics

checkpoint_best_regular.pth (epoch 43) — Per-class AP (detection)

checkpoint_best_ema.pth (epoch 49) — Overall metrics

Per-class AP is not available for checkpoint_best_ema.pth. Only per-class metrics were logged for the regular checkpoint during training.

Note on class imbalance: the model performs well on structural zones (main text, margin text, graphic zones, numbering) but is substantially weaker on the smaller formula classes (-inline, -strikethrough).

How to Get Started with the Model

from rfdetr import RFDETRSegNano
import cv2
from rfdetr.assets.coco_classes import COCO_CLASSES

model = RFDETRSegNano(pretrain_weights="checkpoint_best_regular.pth")

image = cv2.imread("image.jpg")

detections = model.predict(image, threshold=0.4)

labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]

annotated_image = sv.MaskAnnotator().annotate(detections.metadata["source_image"], detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)

cv2.imwrite("image_annotated.jpg", annotated_image)

Replace "image.jpg" with the path to your manuscript image.

The threshold can be changed, but we noticed that we have fair predictions with a value of 0,4. Under this value we start having overlapping regions and over this value we loose some correct predictions with low confidence score.

eScriptorium Pipeline

A preprocessing pipeline for preparing model outputs for import into eScriptorium is available here:

Pipeline RF-DETR-Seg to XML PAGE

It converts this model's zone predictions into the XML-Page format compatible with eScriptorium.

Limitations

• Fine-tuned on manuscripts from a single author (16th–early 17th c.). Generalization to other authors or periods has not been tested.
• Performance depends on image quality.
• GraphicZone-formula is a generic label for formulae that weren't sorted into one of the three more specific classes (-complex, -inline, -strikethrough). Since annotations came from different practices and different annotators, not all formulae were manually reclassified. This is a known issue we plan to fix in a future version of the dataset.

Funding

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, Advanced Grant ADG No. 101020985.