receipt-qwen25vl-3b-lora

Receipt Qwen2.5-VL 3B LoRA

This repository contains a LoRA adapter for receipt image information extraction with Qwen/Qwen2.5-VL-3B-Instruct as the base model.

The intended task is:

receipt image -> strict receipt_extraction.v1 JSON

This is an early public bootstrap model. It is useful for testing a receipt-to-JSON workflow, but it is not production-ready for Korean ERP evidence automation without additional in-domain Korean receipt labels.

Base Model

• Base: Qwen/Qwen2.5-VL-3B-Instruct
• Adapter type: LoRA / PEFT
• Training method: SFT with TRL

Training Data

The training set was built from public receipt key-information-extraction datasets.

Dataset build summary:

The training target schema was normalized to:

{
  "schema_version": "receipt_extraction.v1",
  "document_type": "CARD_RECEIPT|GENERAL_RECEIPT|TAX_INVOICE|INVOICE|UNKNOWN",
  "merchant_name": "string|null",
  "business_no": "string|null",
  "transaction_date": "YYYY-MM-DD|null",
  "transaction_time": "HH:MM:SS|null",
  "approval_no": "string|null",
  "receipt_no": "string|null",
  "card_no_masked": "string|null",
  "supply_amount": "number|null",
  "vat_amount": "number|null",
  "total_amount": "number|null",
  "currency": "string|null",
  "address": "string|null",
  "phone_number": "string|null",
  "items": []
}

Training Run

Preliminary Evaluation

Evaluation was run on the first 20 validation examples from nanonets/key_information_extraction.

The metric is strict field-level exact match after light scalar normalization. This is intentionally harsh for addresses and formatting-heavy fields. It is still useful for seeing whether the adapter is actually improving the fields required by ERP evidence processing.

Critical fields used for the comparison:

merchant_name, business_no, transaction_date, receipt_no,
vat_amount, total_amount, address, phone_number

Field-Level Exact Match

Interpretation

This adapter improved strict JSON compliance and schema discipline, but the preliminary test shows that it is not yet better than the base model on several critical receipt fields.

Observed behavior:

• Better: strict JSON output, schema consistency, total amount extraction.
• Worse than base in this preliminary sample: transaction date, receipt number, VAT amount, phone number.
• Main likely cause: public bootstrap labels are sparse and not aligned with Korean ERP receipt requirements. Many important ERP fields are missing or inconsistent across source datasets.

Recommended Next Steps

For practical Korean ERP receipt extraction, use this model as a bootstrap only.

Recommended improvement path:

1. Label 300-1,000 real Korean business receipt images with the target JSON schema.
2. Include common receipt types: corporate card receipts, delivery receipts, fuel, meals, lodging, transport, online payment receipts.
3. Add PaddleOCR-VL or another OCR pass as auxiliary input for fields that are hard to read visually.
4. Re-train with stronger LoRA coverage, such as more target modules and a higher rank.
5. Evaluate on a blind Korean receipt set with field-level metrics for merchant_name, business_no, transaction_date, approval_no, supply_amount, vat_amount, and total_amount.

Loading

This repository is a PEFT LoRA adapter. Load it together with the base model:

from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

base_model = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter_model = "hoin1218/receipt-qwen25vl-3b-lora"

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(base_model, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_model)
processor = AutoProcessor.from_pretrained(adapter_model)

Limitations

• Not validated for production Korean receipts yet.
• Public training data is mostly non-Korean.
• Exact extraction of business numbers, approval numbers, VAT, and date fields requires in-domain labels.
• The model can still omit visible values or produce wrong values. Use rule-based validation and human review for accounting workflows.