Falcon-OCR-Poneglyph

Falcon-OCR · Poneglyph

Recognition of French manga speech bubbles, fine-tuned on validated Poneglyph transcriptions. Input: an already cropped bubble. Output: text. No layout detector is trained.

Which model do the numbers describe?

The root model was retrained from the base checkpoint on 100% of the exported corpus. It has no independent held-out score. All test metrics and analysis images below describe the separate evaluated/ checkpoint, before full-data retraining.

Selection uses validation CER only, with early stopping. Test pages are used once after selection, for both the base model and selected checkpoint. Full-data retraining uses the selected number of epochs and restarts from the base, with no further test-based selection. This procedure does not guarantee that full-data retraining beats the evaluated checkpoint.

CER is corpus edit distance divided by corpus reference length (not average sample CER). Normalization is NFC and whitespace collapse; accents, punctuation and case are preserved. benchmark_test.json includes a 95% bootstrap interval resampling whole pages, length/shape slices, per-bubble predictions, and measured generation time. The split is by page, with exact duplicate crops/pages grouped to avoid leakage. Near-duplicate scans and different pages from the same series can remain related; these results do not measure generalization to completely unseen series.

Inference

import torch
from PIL import Image
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Remidesbois/Falcon-OCR-Poneglyph", trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda").eval()
# Use subfolder="evaluated" to load the independently evaluated checkpoint.
texts = model.generate(Image.open("bubble.png").convert("RGB"),
                       category="plain", min_dimension=64,
                       max_dimension=896, max_new_tokens=256)
print(texts[0])

The upstream generation method can round the token budget up to a block boundary. The benchmark uses an exact cap. inference.py provides the benchmark-compatible preprocessing (white padding for pathological aspect ratios), FP32 master weights with BF16 autocast, and exact-cap decoding. Loading all weights directly in BF16 can introduce additional rounding differences. Remote code is the pinned upstream revision 42ec56b72a23984ac059e7c8a6d397a8529423fe; review it before use.

Training

RTX 5090 profile: PyTorch 2.11 / CUDA 13, full parameter fine-tuning, FP32 master weights and AdamW states, BF16 autocast, SDPA with differentiable attention sinks, gradient checkpointing, cosine schedule, warmup and gradient clipping. Only target text and its stop token are supervised; images and prompts are masked. Loss is averaged per bubble. Conservative photometric augmentation applies only in training. The inference-only Triton MLP is replaced by differentiable PyTorch during training. Published weights retain the original Falcon architecture and inference code.

Exported corpus: 9707 bubbles. Selected training duration: 6 epochs. See run_summary.json, run_config.json, dataset_report.json, environment.txt, and training_code/ for provenance. Training data are validated annotations; empty references and invalid/missing crops are listed in the export accounting. Download errors fail the run instead of silently dropping data. Corpus snapshots and their assignment fingerprints are fixed on resume.

Analysis of the evaluated checkpoint

Representative examples (largest absolute errors, followed by exact matches):

Model weights derive from Falcon-OCR. Manga examples retain their respective owners' rights; the model license does not license the source images.