koharu-text-sam-ts-l

Koharu Text SAM-TS-L

koharu-text-sam-ts-l is a ViT-L Hi-SAM/SAM-TS TextSeg model fine-tuned for binary text segmentation on manga pages. It detects text pixels rather than speech balloons, panels, or individual text lines.

The model was fine-tuned from the SAM-TS-L TextSeg checkpoint from Hi-SAM. The released weights use SafeTensors only; no pickle checkpoint is required for the full model.

Files

Use model.safetensors unless you specifically need the smaller adapter state. The full file loads strictly into a freshly constructed SAM-TS-L model and does not require the original SAM or TextSeg .pth files.

Results

Metrics use micro-averaged binary pixel counts. HR refers to the high-resolution SAM-TS output.

The fine-tuned model improved HR IoU on 39 of the 40 test pages. Test HR IoU ranged from 0.5616 to 0.6977 across the four held-out books.

Training data

Training used the masks from Mask Dataset for: Unconstrained Text Detection in Manga: a New Dataset and Baseline by Julian del Gobbo and Rosana Matuk Herrera, licensed CC BY 4.0, paired locally with authorized Manga109 images. Manga109 images are not included in this model repository.

The split was deterministic and book-disjoint:

• 36 training books: 359 full pages
• 5 validation books: 50 full pages
• 4 test books: 40 full pages
• one empty-mask training page was excluded

The training stream contained 1,795 entries: 1,436 native 1024×1024 crops (80%) and 359 full pages resized and padded to 1024×1024 (20%). This model was not trained with the mayocream/manga109-segmentation dataset.

Training configuration

• 20 epochs; best checkpoint at epoch 20
• AdamW, weight decay 0.05
• peak learning rate 1e-5
• one-epoch warmup followed by cosine decay to 1e-6
• batch size 16, BF16
• NVIDIA B300 SXM6 AC
• PyTorch 2.9.1 + CUDA 13.0

Usage

Clone Hi-SAM and install its dependencies, plus huggingface_hub and safetensors. The included command-line example loads the complete SafeTensors state and writes a binary PNG mask:

python inference.py input-page.jpg output-mask.png --hi-sam-root /path/to/Hi-SAM

Equivalent model-loading code:

from types import SimpleNamespace

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hi_sam.modeling.build import model_registry

args = SimpleNamespace(
    checkpoint=None,
    model_type="vit_l",
    attn_layers=1,
    prompt_len=12,
    hier_det=False,
)
model = model_registry["vit_l"](args=args)
weights = hf_hub_download(
    "mayocream/koharu-text-sam-ts-l", "model.safetensors"
)
model.load_state_dict(load_file(weights, device="cpu"), strict=True)
model = model.eval().cuda()

Inputs are RGB tensors in the 0–255 range. Resize the page so its longest side is 1024 pixels, pad the bottom and right sides to 1024×1024 with RGB value 128, and crop away the padding from the predicted mask before resizing it back to the original page size. inference.py implements this exact preprocessing.

Limitations

• Evaluation covers 45 books and only the first ten pages of each book.
• The model is strongest on clearly rendered Japanese manga text. Tiny text, stylized sound effects, dense pages, and text-like artwork remain difficult.
• The output is binary text segmentation. It does not assign instances, recognize characters, or associate text with speech balloons.
• Manga and comic styles outside Manga109 may have a larger domain gap.
• Hi-SAM is custom research code, so this repository is not directly compatible with transformers.AutoModel or the hosted Hugging Face inference widget.

Reproducibility and integrity

The complete SafeTensors file was loaded with strict=True and reevaluated on all validation and test pages. It reproduced the original fine-tuned checkpoint metrics exactly in the local CUDA 13 evaluation environment.

License and attribution

The model is released under Apache License 2.0, matching Hi-SAM and Segment Anything. The training masks are CC BY 4.0 and are attributed above. Use of Manga109 imagery remains subject to the Manga109 terms; no Manga109 images are redistributed here.