rt-detrv2-heron-101-rukopys

rt-detrv2-heron-101-rukopys

An RT-DETRv2-R101 (RTDetrV2ForObjectDetection, ~76.7M params) document-layout detector for handwritten Ukrainian pages, fine-tuned on the Rukopys dataset with a 7-class head: 0 handwritten · 1 printed · 2 formula · 3 table · 4 annotation · 5 image · 6 graph.

TL;DR

Intended use

Detecting and classifying page regions on handwritten Ukrainian documents, upstream of region recognizers (e.g. Hukyl/trocr-large-rukopys for text, Hukyl/trocr-base-rukopys-formula for formulas). For a stronger single detector on the same task, see Hukyl/doclayout-yolov10-rukopys.

How to use

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForObjectDetection

repo = "Hukyl/rt-detrv2-heron-101-rukopys"
model = AutoModelForObjectDetection.from_pretrained(repo).eval()
processor = AutoImageProcessor.from_pretrained(repo)

image = Image.open("page.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# threshold low — the scores are diffuse; max-F1 is at ~0.075 (see Limitations)
target_sizes = torch.tensor([image.size[::-1]])  # (height, width)
result = processor.post_process_object_detection(
    outputs, threshold=0.1, target_sizes=target_sizes
)[0]
for score, label, box in zip(result["scores"], result["labels"], result["boxes"]):
    print(model.config.id2label[label.item()], round(score.item(), 3), box.tolist())

Files

model.safetensors, config.json,   the shipped checkpoint — loads directly
preprocessor_config.json          with from_pretrained
training_meta.json                recorded recipe + the shipped checkpoint's metrics
metrics.json                      val metrics, aggregate + per-class
selection_comparison.json         the three checkpoint-selection axes side by side
training_log.jsonl                per-epoch loss / mAP / P / R curves

Head re-initialisation (17 → 7)

The backbone, encoder, decoder, and the class-agnostic box-regression head load from the base checkpoint as-is; the classification head is re-initialised to the 7 Rukopys classes, with id2label/label2id baked into the config.

Training

Fine-tuned on the human-labeled Rukopys train split via the 🤗 Trainer. The image processor applies a deterministic 640×640 resize with ImageNet normalisation.

Hyperparameters (as launched)

The full recorded recipe ships in training_meta.json; per-epoch curves in training_log.jsonl.

Checkpoint selection

The run tracked three selection axes; val loss and val mAP@50 are anti-correlated across this run (Pearson ≈ −0.78). This repo ships the mAP@50-selected checkpoint.

Class distribution (region counts)

Results

Held-out val split of the Rukopys train data: 133 pages / 2,629 regions (stratified holdout, seed 42). mAP is threshold-free with box matching at IoU 0.50 (mAP@50–95 averages IoU 0.50:0.95); precision and recall are reported at a fixed confidence ≥ 0.50 operating point (class-aware greedy matching at IoU ≥ 0.50). The model's per-class sigmoid scores are diffuse and few boxes clear 0.50, so recall at this operating point collapses to ~0.01 — an artifact of the threshold, not the miss rate; the threshold-free mAP is the signal. Per-class precision/recall are not recorded.

Aggregate (macro over 7 classes)

Per class

We also acknowledge that printed/table/annotation/image/graph n is quite small, so measuring detection metrics against them is quite noisy.

Limitations

• Confidence is uncalibrated: the empirical max-F1 confidence is ≈0.075, far below a typical 0.25–0.50 first guess, and F1 collapses by conf 0.30. Tune the inference threshold against your own target metric.
• annotation (mAP@50 0.32) is a weak class.
• Trained and evaluated at 640×640; very small or dense regions on high-resolution scans may benefit from a higher inference resolution.
• Handwritten Ukrainian school/archival-style pages only; behaviour on other document types is untested.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

Model weights are Apache-2.0, inherited from RT-DETRv2 and the base checkpoint.