manga-text-detector-v0

manga-text-detector-v0

A YOLO11s detector that draws one box per text region on a manga page — one box per speech bubble, caption, or free-floating line of dialogue — so each region can be cropped and handed to a text-extraction model.

It is the winner of a five-backend bake-off run on Manga109-s, and on 20 held-out books it reaches 95.6% extraction-ready regions, against a 99.7% ceiling. It beats comic-text-detector by +7.2 points (paired book-level bootstrap, 95% CI [4.8, 9.2], winning 16 of 20 books) and the shipped Google Cloud Vision + region-merge pipeline it was built to replace by +21.7 points — at 9.43M parameters and 19MB.

Why not mAP

The downstream job is cropping, not localization, so this model is not ranked on IoU-based metrics. The ranking metric is ready: the fraction of ground-truth regions whose crop is extraction-ready — the box contains the whole region and contains nothing from any other region. A prediction fails in one of five ways: trunc (clips the region), merge (swallows a neighbouring region), split (region covered by several boxes), miss, or it is ready.

This distinction is not cosmetic. Dilating every predicted box by 10px lifts ready from 0.1% to 78.8% while F1 falls from 0.927 to 0.268. Fat boxes are strictly better for cropping and strictly worse by IoU, so any F1/mAP comparison of text-region detectors ranks them on the wrong axis. For reference only, the Ultralytics validation numbers at the final epoch (both classes, page-level val split) are mAP50 0.915, mAP50-95 0.739, precision 0.917, recall 0.864 — reported for comparability, not used for any decision in this project.

The ceiling is 99.7%, not 100%: feeding ground truth back in as a prediction still leaves ~0.3% of regions unready, because some annotated regions genuinely overlap and contaminate each other's crops. Read every number below against 99.7%, not 100%.

Results

20 held-out books, 80 spreads, 1,341 ground-truth regions. Each backend at its own best padding setting, since dilation is free downstream and comparing raw boxes just rewards whoever draws loosest. CIs are book-level bootstrap.

Two things about that table deserve emphasis, one in each direction:

• The comic-text-detector row is not a clean number. CTD trained on ~4,300 Manga109-s images out of a corpus of 8,519 spreads — roughly half — and upstream publishes no book list or split, so an uncontaminated rescore is impossible. An expected ~half of the evaluation books were seen during its training. This model is book-disjoint by construction, so the +7.2 points is a clean number measured against a dirty one, and the real gap is likely wider.
• Holdout numbers are not corpus numbers. Every backend scores 2-3 points higher on these 20 books than corpus-wide (the ceiling too: 99.7% vs 99.3%). Ranking is unchanged but absolute values do not transfer. On the full 348-spread corpus-wide test set, where this model cannot be scored without leakage, CTD is at 86.5% and GCV+merge at 71.0%.

Where the residual 4.1 points goes

split 3.0% + miss 1.9% + trunc 2.2%, and merge only 0.4% — seven times below CTD's 2.8%. The failure shape inverts against every other backend tested: the incumbents are truncation-dominated and CTD is merge-dominated, while this model's problem is purely completeness. Instance separation is effectively solved, and generous NMS (IoU 0.7) on region-level labels turned out to be enough for it — an NMS-free architecture is not what this needs next.

Padding: read this before using the model

The boxes are meant to be dilated before cropping, and how you dilate matters more than any inference-time knob.

Padding is measured in units of each box's own short side (≈ one character height for vertical Japanese text), and the policy that wins for every backend tested is contact: dilate each side up to a cap, but stop halfway to the nearest predicted neighbour. An isolated bubble gets the full dilation; a bubble packed against three others gets only the gap. Uniform padding has to trade truncation against merging with a single cap for the whole page; contact-limiting decouples them, and is worth 9-18 points over uniform for every backend.

Best setting for this model: contact padding at a 30-40% cap (95.6% ready, vs 92.0% at its best uniform setting). Unbounded contact padding loses — with nothing to collide with, a box grows into whatever the detector missed, and a missed region cannot block it.

Confidence threshold

Use conf 0.05, not a conventional 0.25-0.5. Performance is monotonic in the wrong direction from the usual instinct: 95.6% at conf 0.05 declining to 93.0% at 0.6. Under contact padding every prediction is also a blocker that stops its neighbours from over-expanding, so a spurious box costs one wasted crop but protects the boxes around it, while a missed box costs its own region and lets its neighbour grow into the gap. Instance recall outranks boundary precision here.

The observed optimum sits at the 0.05 floor used when dumping predictions, so the true optimum may be lower still — untested. NMS IoU was swept over {0.4 … 0.9} and is flat (95.8-96.1%); leave it at 0.7 and don't spend time on it.

Usage

from ultralytics import YOLO
from PIL import Image, ImageOps

model = YOLO("best.pt")

page = Image.open("page.jpg")
page = ImageOps.exif_transpose(page)   # see "Honour EXIF" below — this line matters

r = model.predict(page, imgsz=1024, conf=0.05, iou=0.7, verbose=False)[0]

boxes = [b.xyxy[0].tolist() for b in r.boxes if int(b.cls) == 0]   # class 0 = text-region

Then apply contact padding at a 30-40% cap over boxes before cropping.

Three input requirements, each of which cost accuracy when violated during evaluation:

1. One page per call, not a double-page spread. The model was trained on single pages split from Manga109 spreads at the gutter (this doubles effective resolution for the same compute, and matches how a phone camera sees a page). Feeding a full ~1654×1170 spread halves the resolution the model sees on each page. If your source is spreads, split at the midline first — only 5 regions in 94,055 straddle the gutter, so this costs essentially nothing.
2. Honour EXIF orientation before inference. A test photo that looked 90°-rotated in preview decoded upright via its EXIF tag, and the model's output differed sharply between the two. A pipeline that ignores EXIF silently loses accuracy on exactly the images real users capture.
3. Upright pages only — see limitations.

TFLite and ONNX exports are included and load through the same ultralytics API (YOLO("best-1024-fp32.tflite", task="detect")). All four exports were scored through the identical ready harness as the .pt checkpoint and tie at 96.1% on the dev set at conf 0.05 — neither export nor int8 quantization costs anything at this operating point.

On-device (Pixel 10a)

Benchmarked with Google's official android_aarch64_benchmark_model.apk.

Ship on the GPU delegate with fp32. That single choice is a bigger lever than quantization, a resolution cut, or distillation — none of which were needed. Notes:

• CPU is not viable at any precision. 8 threads is slower than 4 on this big.LITTLE SoC.
• int8 + GPU delegate is a hard failure, not a slow path: the OpenCL backend rejects the int8 ops, falls back to OpenGL, fails to prepare, and runs zero inferences. The two speedups do not stack. int8 is worth keeping for CPU-only fallback devices and bundle size, not for the GPU path.
• The GPU delegate pays a ~11 s shader-compile init on every launch — measured twice back to back with no improvement, so there is no OS-level shader cache to rely on. An app that wants a fast second launch must opt into TFLite's TfLiteGpuDelegateOptionsV2.serialization_dir explicitly.
• ONNX Runtime was evaluated as an alternative (accuracy identical) and its best number on this device is CPU/4-threads at 491-588 ms — 5-6x slower, with no path to the GPU delegate TFLite uses.

Training

Everything about v0 is deliberately stock — the point was to establish the pipeline and find out which failure mode dominates, not to be clever.

Three decisions that shaped the result more than any hyperparameter:

• Region-level labels, no line-level intermediate and no grouping stage. Manga109's <text> boxes are already at the granularity the metric scores. The two OCR-based incumbents both need a separate fragment→region grouping algorithm on top, and that grouping is worth ~20 points to them — but a detector trained directly on regions beats the whole pipeline.
• SFX is a second class, not masked out. COO onomatopoeia annotations cover 84% of training spreads (33,573 boxes) — much better coverage than "a subset" suggests. Explicit supervision on the most confusable category beats hiding it. SFX is out of scope at scoring time (ignore regions), but the model is told what it is.
• Not a comic-text-detector fine-tune. CTD already trained on ~half of Manga109-s, so most of that gradient signal is already absorbed and any gain measured on Manga109 would be doubly contaminated.

Label quality was audited before training rather than assumed (94,661 regions across the 67 training books): 606 boxes dropped by the SFX overlap rule, 1,056 overlapping pairs of which 977 overlap by <25% of the smaller box (adjacent vertical columns grazing — harmless), leaving 10 truly nested regions in 94,055. 15 boxes under 12px on the short side, 0 degenerate. Label noise is not what stands between this model and the ceiling.

Limitations

• Rotation-sensitive. On a single photo of a printed page: upright, every bubble boxed at mean confidence 0.93; rotated 180°, 12 boxes at 0.85; rotated 90°, only 9 of 11 at 0.81. Vertical Japanese inverted is still vertical columns, but a quarter-turn makes them horizontal. Fix this upstream — the device knows its own orientation — rather than with training augmentation.
• Capture-condition evidence is thin. The model survived curvature, glare, print halftone, a thumb in frame and an intruding facing page on one photo. One photo is not evidence; there is no ground truth for it and the recall counts above were eyeballed from overlays. Manga109 is clean scans, and the gap between benchmark scans and phone photos is the least-measured thing about this model.
• Japanese manga only. Trained exclusively on Manga109-s: vertical Japanese text, B&W screentone artwork, Japanese bubble conventions. Western comics, colour pages, webtoons and horizontal-script languages are all out of distribution and untested.
• It detects, it does not read. No text is produced — only geometry. Crops go to a separate extraction model.
• sfx (class 1) is untuned and unscored. It exists to keep the model from confusing onomatopoeia with dialogue. Its accuracy has never been measured; do not rely on it.
• The 4.1-point residual is real. ~2 regions per spread still come out unready — mostly split or missed, not merged. This is good enough to be worth shipping and not good enough to be unsupervised.

Data and licence

Trained on Manga109-s, the redistributable 87-book subset of Manga109, used under its academic-use terms. Manga109-s images are not redistributable and none are included here — only trained weights. The model is released CC BY-NC-SA 4.0 to stay consistent with the spirit of that corpus's terms; if you need a commercial licence for a manga text detector, train on data you have rights to.

Manga109 must be cited if you use this model in published work:

@article{mtap_matsui_2017,
    author={Matsui, Yusuke and Ito, Kota and Aramaki, Yuji and Fujimoto, Azuma and Ogawa,
            Toru and Yamasaki, Toshihiko and Aizawa, Kiyoharu},
    title={Sketch-based Manga Retrieval using Manga109 Dataset},
    journal={Multimedia Tools and Applications},
    volume={76}, number={20}, pages={21811--21838}, year={2017}
}
@article{multimedia_aizawa_2020,
    author={Aizawa, Kiyoharu and Fujimoto, Azuma and Otsubo, Atsushi and Ogawa, Toru and
            Matsui, Yusuke and Tsubota, Koki and Ikuta, Hikaru},
    title={Building a Manga Dataset ``Manga109'' with Annotations for Multimedia
           Applications},
    journal={IEEE MultiMedia}, volume={27}, number={2}, pages={8--18}, year={2020}
}

Files