paddleocr-vl-1.6-manga-lora

PaddleOCR-VL-1.6 manga crop reader

A crop-level reader for manga text, fine-tuned from PaddleOCR-VL-1.6 on its OCR: task. It reads sound effects (hand-lettered, vertical or diagonal, with hearts and small kana) and speech balloons in Japanese, Korean and Chinese. A multi-line block comes back with \n between its lines.

It reads crops, not pages: pair it with a text detector (see Detection).

Files

Usage

Both v6 files are needed: merge the adapter into the base, then load the LM weights over it.

import torch
from PIL import Image
from huggingface_hub import snapshot_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoModelForImageTextToText, AutoProcessor

base = "PaddlePaddle/PaddleOCR-VL-1.6"
d = snapshot_download("sorryhyun/paddleocr-vl-1.6-manga-lora", allow_patterns="v6/*") + "/v6"

model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, attn_implementation="sdpa")
model = PeftModel.from_pretrained(model, d).merge_and_unload()
missing = model.load_state_dict(load_file(f"{d}/lm.safetensors"), strict=False)
assert not missing.unexpected_keys, missing.unexpected_keys[:5]
model = model.cuda().eval()
proc = AutoProcessor.from_pretrained(base)

msgs = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "OCR:"}]}]
prompt = proc.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
size = {"shortest_edge": proc.image_processor.size["shortest_edge"], "longest_edge": 1280 * 28 * 28}

@torch.inference_mode()
def read(crops, batch_size=16):
    """PIL crops in, strings out (input order)."""
    order = sorted(range(len(crops)), key=lambda i: crops[i].width * crops[i].height)
    out = [""] * len(crops)
    for s in range(0, len(order), batch_size):
        idx = order[s : s + batch_size]
        inputs = proc(text=[prompt] * len(idx), images=[crops[i] for i in idx], padding=True,
                      padding_side="left", return_tensors="pt", images_kwargs={"size": size}).to("cuda")
        gen = model.generate(**inputs, max_new_tokens=80, do_sample=False, use_cache=True)
        n = inputs["input_ids"].shape[-1]
        for i, row in zip(idx, gen):
            out[i] = proc.tokenizer.decode(row[n:], skip_special_tokens=True).strip()
    return out

crop = Image.open("sfx_crop.png").convert("RGB")  # one text block, ~12 % margin, upright
print(read([crop])[0])

• Pass use_cache=True: the base's generation_config.json turns the cache off.
• Hearts come out as ♥.
• To join a block onto one line, join Japanese and Chinese on "" and Korean or English on " ".
• Like the base, it can repeat a glyph without stopping on a few percent of short crops (ぐくーーーー…). Cap the output length or add a repetition guard.

Results

Greedy decoding, exact match after NFKC and glyph folding, whitespace and line breaks ignored.

* scored on an earlier crop cut. For reference, the published COO baseline (TRBA + 2D attention, full test set) is 81.2 %.

Training

Detection

Box the text with deepghs/AnimeText_yolo (yolo12l_animetext/model.onnx), then crop each box grown by 12 % of its longer side. Settings:

The detector is GPL-3.0 and trained on CC-BY-NC-SA-4.0 data; it is not part of this repo.

import cv2, numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

onnx = hf_hub_download("deepghs/AnimeText_yolo", "yolo12l_animetext/model.onnx")
sess = ort.InferenceSession(onnx, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

def detect(bgr, imgsz=640, conf=0.25, nms=0.5, nest=0.85):
    h0, w0 = bgr.shape[:2]
    r = min(imgsz / h0, imgsz / w0)
    nw, nh = max(1, round(w0 * r)), max(1, round(h0 * r))
    canvas = np.full((imgsz, imgsz, 3), 114, np.uint8)
    canvas[:nh, :nw] = cv2.resize(bgr, (nw, nh), interpolation=cv2.INTER_AREA)
    x = np.ascontiguousarray(canvas[:, :, ::-1].transpose(2, 0, 1)[None].astype(np.float32) / 255)
    pred = sess.run(None, {sess.get_inputs()[0].name: x})[0][0].T  # (N, 5): cx cy w h score
    p = pred[pred[:, 4] >= conf]
    if not len(p):
        return []
    xywh = np.stack([p[:, 0] - p[:, 2] / 2, p[:, 1] - p[:, 3] / 2, p[:, 2], p[:, 3]], 1)
    keep = np.asarray(cv2.dnn.NMSBoxes(xywh.tolist(), p[:, 4].tolist(), conf, nms)).flatten()
    boxes = []
    for i in keep:
        x, y, w, h = xywh[i] / r
        x0, y0, x1, y1 = max(int(x), 0), max(int(y), 0), min(int(x + w), w0), min(int(y + h), h0)
        if x1 > x0 and y1 > y0:
            boxes.append((x0, y0, x1, y1))
    def inside(a, b):  # share of a's area inside b
        ix = max(0, min(a[2], b[2]) - max(a[0], b[0])); iy = max(0, min(a[3], b[3]) - max(a[1], b[1]))
        return ix * iy / max(1, (a[2] - a[0]) * (a[3] - a[1]))
    area = lambda b: (b[2] - b[0]) * (b[3] - b[1])
    return [b for b in boxes
            if sum(area(o) < area(b) and inside(o, b) >= nest for o in boxes if o is not b) < 2]

def crop(bgr, box, pad=0.12):
    x0, y0, x1, y1 = box
    g = round(pad * max(x1 - x0, y1 - y0))
    h, w = bgr.shape[:2]
    return Image.fromarray(bgr[max(y0 - g, 0):min(y1 + g, h), max(x0 - g, 0):min(x1 + g, w), ::-1])

page = cv2.imread("page.png")
texts = read([crop(page, b) for b in detect(page)])  # read() from Usage

anime_tools packages the detector, this reader, a repetition guard and reading order: python -m anime_tools.stages.cli.ocr_captions.

Data attribution

Trained on Manga109-s and the COO (Comic Onomatopoeia) annotations, plus Korean / Chinese crops from fan-translated pages on gelbooru and Korean fan-translated doujinshi. No images are redistributed here. Manga109 comics are courtesy of their respective authors (see the Manga109 site). If you use this model in academic work, please cite:

@inproceedings{baek2026mangav26,
  title     = {{Manga109-v2026: Revisiting Manga109 Annotations for Modern Manga Understanding}},
  author    = {Baek, Jeonghun and Miyai, Atsuyuki and Onohara, Shota and Ikuta, Hikaru and Aizawa, Kiyoharu},
  booktitle = {ICML Workshop},
  year      = {2026},
}
@article{multimedia_aizawa_2020,
  author={Kiyoharu Aizawa and Azuma Fujimoto and Atsushi Otsubo and Toru Ogawa and Yusuke Matsui and Koki Tsubota and Hikaru Ikuta},
  title={Building a Manga Dataset ``Manga109'' with Annotations for Multimedia Applications},
  journal={IEEE MultiMedia}, volume={27}, number={2}, pages={8--18}, doi={10.1109/mmul.2020.2987895}, year={2020}
}
@article{mtap_matsui_2017,
  author={Yusuke Matsui and Kota Ito and Yuji Aramaki and Azuma Fujimoto and Toru Ogawa and Toshihiko Yamasaki and Kiyoharu Aizawa},
  title={Sketch-based Manga Retrieval using Manga109 Dataset},
  journal={Multimedia Tools and Applications}, volume={76}, number={20}, pages={21811--21838}, doi={10.1007/s11042-016-4020-z}, year={2017}
}
@inproceedings{baek2022coo,
  title     = {COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts},
  author    = {Baek, Jeonghun and Matsui, Yusuke and Aizawa, Kiyoharu},
  booktitle = {ECCV},
  year      = {2022},
}

License

Weights: Apache-2.0, as the base model. The training data is governed by the Manga109-s terms (no redistribution of images; models may be published with attribution).