lfm2.5-vl-450m-extract-ja-receipt-lora

LFM2.5-VL-450M-Extract — Japanese Receipt KIE LoRA

LoRA adapter that teaches LiquidAI/LFM2.5-VL-450M-Extract to extract structured fields from Japanese receipts (店名・日付・時刻・合計・税額・品目リスト).

The base model is a strong structured-extraction VLM but struggles severely with Japanese (CJK) text. This adapter fixes that for the receipt domain — trained in 41 minutes on 3× RTX 3090.

Results

Evaluated on 200 held-out real receipts from llm-jp/jawildtext receipt_kie (never seen in training). Exact match / character similarity:

On out-of-distribution real receipt photos the base model frequently hallucinates totals and degenerates into repetition loops; with this adapter, totals / dates / item structure become reliable.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "LiquidAI/LFM2.5-VL-450M-Extract"
processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(model, "outxci/lfm2.5-vl-450m-extract-ja-receipt-lora").merge_and_unload()

SYSTEM = """store_name: The name of the store as printed on the receipt
date: The purchase date as printed on the receipt
time: The purchase time as printed on the receipt
total_amount: The total amount paid
tax_amount: The tax amount
items: List of purchased items with name, price and quantity"""

image = Image.open("receipt.jpg").convert("RGB")
conv = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
    {"role": "user", "content": [{"type": "image", "image": image}]},
]
inputs = processor.apply_chat_template(conv, add_generation_prompt=True,
            return_tensors="pt", return_dict=True, tokenize=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Output is a single JSON object:

{"store_name": "...", "date": "2020年4月24日(金)", "time": "19:10",
 "total_amount": "¥1,187", "tax_amount": "¥93",
 "items": [{"name": "...", "price": "108", "qty": ""}]}

Training recipe

• Data: llm-jp/jawildtext receipt_kie (951 real mobile-captured receipts with field-level GT, Apache 2.0) × 6 epochs, mixed with 4,000 synthetic Japanese text-line OCR samples from deepcopy/japanese-synthetic-ocr-150k (upstream Kaggle source jp-font-image-dataset-02 states CC0) to retain glyph-reading ability
• LoRA: r=32, α=64, dropout 0.05 — applied to language model (q/k/v/out_proj, w1/w2/w3, conv in_proj) and vision tower (q/k/v/out_proj, fc1/fc2); multi-modal projector fully trained (25.7M trainable params, 5.4%)
• Schedule: effective batch 12 (3 GPU × grad-accum 4), AdamW lr 1e-4, cosine w/ 3% warmup, bf16
• Hardware: 3× RTX 3090, DDP, 41 min
• Held-out split: last 200 rows of receipt_kie reserved for evaluation

A two-stage approach was validated first: training on text-line OCR alone improves glyph reading but does not transfer to receipts — task-format data (image → fields JSON) was the key.

Limitations

• store_name remains hard (logos / decorative fonts): 10% exact match
• Item names are approximate — kana-level misreads remain (e.g. イタチョコ → イチャコ), a resolution limit of the 450M-scale vision tower
• The model sometimes fills fields that are not visible in the image instead of leaving them empty
• Trained and evaluated on Japanese receipts only; behaviour on other document types follows the base model

License

This adapter is a Derivative Work of LFM2.5-VL-450M-Extract and is distributed under the LFM Open License v1.0 (see LICENSE). Commercial use is permitted for entities below the license's annual-revenue threshold; see the license text for details.