qwen2-vl-2b-cord-receipts-lora

Qwen2-VL-2B — CORD Receipt Extraction (LoRA)

A LoRA adapter for Qwen/Qwen2-VL-2B-Instruct, fine-tuned to extract structured data from receipts as JSON (line_items, subtotal, tax, total).

Part of the Vision-Agent project — an LLM-orchestrated multi-agent receipt understanding system.

Results (CORD test set)

On in-domain extraction this fine-tuned 2B model matches and exceeds a frontier API (Claude Opus 4.8) zero-shot, at ~13× lower inference cost:

Note: this is an in-domain comparison (fine-tuned on CORD, tested on CORD's held-out split; the frontier model is zero-shot).

Usage

import torch
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
from peft import PeftModel

BASE = "Qwen/Qwen2-VL-2B-Instruct"
ADAPTER = "Ji-Qing/qwen2-vl-2b-cord-receipts-lora"  # this repo

processor = AutoProcessor.from_pretrained(BASE)
base = Qwen2VLForConditionalGeneration.from_pretrained(BASE, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, ADAPTER).eval()

INSTRUCTION = (
    "Extract the receipt as JSON with keys: line_items (a list of {name, price}), "
    "subtotal, tax, total. Copy values exactly as printed. Use null if a field is "
    "absent. Respond with only the JSON object."
)
messages = [{"role": "user", "content": [
    {"type": "image", "image": your_pil_image},
    {"type": "text", "text": INSTRUCTION},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[your_pil_image], return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=256)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Training

• Base: Qwen2-VL-2B-Instruct
• Method: LoRA (r=16, alpha=32; targets q/k/v/o_proj) via PEFT
• Data: CORD-v2 train split (~800 receipts), label-masked instruction tuning (loss on the target JSON only)
• Compute: single A100 (~26 min) on Modal

Limitations

• Fine-tuned on CORD (Indonesian receipts); generalization to other formats not evaluated.
• The comparison above is in-domain vs a zero-shot frontier model.