PaddleOCR-VL-For-Manga-GGUF

PaddleOCR-VL-For-Manga GGUF

This repo contains BF16 quantized GGUF files of jzhang533's PaddleOCR-VL-For-Manga model.

It can be run via llama.cpp with the following parameters:

llama-server \
  -m ./PaddleOCR-VL-For-Manga-GGUF/PaddleOCR-VL-For-Manga-BF16.gguf
  --mmproj ./PaddleOCR-VL-For-Manga-GGUF/PaddleOCR-VL-For-Manga-mmproj-BF16.gguf
  --host 0.0.0.0
  --n-gpu-layers 999
  --port 9999
  -c 32768
  --temp 0

Send an image along with the prompt OCR: to receive an OCR response. If you enounter this error: tokenize: error: number of bitmaps (1) does not match number of markers (0). send <__media__>OCR: as your prompt and see if that resolves the issue.

Original discussion for PaddleOCR-VL support in llama.cpp can be found here.

Original model card for PaddleOCR-VL-For-Manga is copied below:

PaddleOCR-VL-For-Manga

<div align="center" style="line-height: 1;"> <a href="https://huggingface.co/jzhang533/PaddleOCR-VL-For-Manga" target="_blank" style="margin: 2px;"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Weights-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/spaces/jzhang533/paddleocr-vl-for-manga-demo" target="_blank" style="margin: 2px;"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DEMO-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://github.com/jzhang533/PaddleOCR-VL-For-Manga" target="_blank" style="margin: 2px;"> <img alt="Github" src="https://img.shields.io/badge/GitHub-Repository-000?logo=github&color=0000FF" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://pfcc.blog/posts/paddleocr-vl-for-manga" target="_blank" style="margin: 2px;"> <img alt="Tutorial" src="https://img.shields.io/badge/👐%20Blog-Tutorial-A5de54" style="display: inline-block; vertical-align: middle;"/> </a> </div>

Model Description

PaddleOCR-VL-For-Manga is an OCR model enhanced for Japanese manga text recognition. It is fine-tuned from PaddleOCR-VL and achieves much higher accuracy on manga speech bubbles and stylized fonts.

This model was fine-tuned on a combination of the Manga109-s dataset and 1.5 million synthetic data samples. It showcases the potential of Supervised Fine-Tuning (SFT) to create highly accurate, domain-specific VLMs for OCR tasks from a powerful, general-purpose base like PaddleOCR-VL, which supports 109 languages.

This project serves as a practical guide for developers looking to build their own custom OCR solutions. You can find the training code at the Github Repository, a step by step tutorial is avaiable here.

Performance

The model achieves a 70% full-sentence accuracy on a test set of Manga109-s crops (representing a 10% split of the dataset). For comparison, the original PaddleOCR-VL on the same test dataset achieves 27% full sentence accuracy.

Common errors involve discrepancies between visually similar characters that are often used interchangeably, such as:

• ！？ vs. !? (Full-width vs. half-width punctuation)
• ＯＫ vs. ok (Full-width vs. half-width letters)
• １２０５ vs. 1205 (Full-width vs. half-width numbers)
• “人” (U+4EBA) vs. “⼈” (U+2F08) (Standard CJK Unified Ideograph vs. CJK Radical)

The prevalence of these character types highlights a limitation of standard metrics like Character Error Rate (CER). These metrics may not fully capture the model's practical accuracy, as they penalize semantically equivalent variations that are common in stylized text.

Examples

How to Use

You can use this model with the transformers, PaddleOCR, or any library that supports PaddleOCR-VL to perform OCR on manga images. The model architecture and weights layout are identical to the base model.

If your application involves documents with structured layouts, you can use your fine-tuned OCR model in conjunction with PP-DocLayoutV2 for layout analysis. However, for manga, the reading order and layout are quite different.

Training Details

• Base Model: PaddleOCR-VL
• Dataset:
  • Manga109-s: 0.1 million randomly sampled text-region crops (not full pages) were used for training (90% split); the remaining 10% crops were used for testing.
  • Synthetic Data: 1.5 million generated samples.
• Training Frameworks:
  • transformers and trl
• Alternatives for SFT:
  • ERNIEKit
  • ms-swift

Acknowledgements

• Manga109-s dataset, which provided the manga text-region crops used for training and evaluation.
• PaddleOCR-VL, the base Visual Language Model on which this model is fine-tuned.
• manga-ocr, used in this project for data processing and synthetic data generation; it also inspired practical workflows and evaluation considerations for manga OCR.

License

This model is licensed under the Apache 2.0 license.