qwen3vl-4b-receipt-extraction-gptq

Qwen3-VL-4B Receipt Extraction (GPTQ 4-bit)

Reads a receipt image and returns it as structured JSON. Calibrated 4-bit GPTQ build, the one this project recommends for GPU serving.

At a glance

Intended use

Turning a photo or scan of a retail receipt into structured JSON, for bookkeeping and expense pipelines, or as a starting point for research on document understanding.

Out of scope

Invoices, handwritten notes, and non-receipt images. The model was never trained on them. It also should not be trusted to make financial decisions on its own; treat the output as an extraction to be reviewed.

How to use

The model expects one image and this exact prompt, which is the prompt it was trained with:

Extract the receipt from the image into a structured JSON. Your output should contain ONLY correct JSON!

Needs GPTQModel (plain transformers would need optimum).

import torch
from gptqmodel import GPTQModel
from transformers import AutoProcessor

REPO = "<your-hf-username>/qwen3vl-4b-receipt-extraction-gptq"
model = GPTQModel.load(REPO, device_map="auto").model   # .model is the underlying HF model
processor = AutoProcessor.from_pretrained(REPO, max_pixels=1600 * 28 * 28)

messages = [{"role": "user", "content": [
    {"type": "image", "image": "receipt.jpg"},
    {"type": "text", "text": "Extract the receipt from the image into a structured JSON. Your output should contain ONLY correct JSON!"},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

It also serves under vLLM, which reads the same GPTQ weight format.

Output

A single JSON object with three top-level keys: menu (the line items), sub_total and total. Every value is a string, copied from the receipt.

{"menu": [{"nm": "Es Teh Manis", "cnt": "2", "price": "8,000"}],
 "sub_total": {"subtotal_price": "8,000"},
 "total": {"total_price": "8,000", "cashprice": "10,000", "changeprice": "2,000"}}

Evaluation

CORD-v2 test split, 100 receipts, greedy decoding, images capped at 1600 patches.

Measured in section 7.2 of finetune_qwen3vl.ipynb.

• field_f1 flattens the predicted and gold JSON into (field-path, value) leaf pairs and scores precision/recall over them. It asks whether the right values landed at the right paths.
• nTED (normalized tree-edit distance, following the CORD paper) compares the two JSON trees directly. It catches structural mistakes, like wrong nesting or a missing key, that leaf-flattening misses.

Neither subsumes the other, which is why both are here.

Training

Fine-tuned from Qwen/Qwen3-VL-4B-Instruct with LoRA on naver-clova-ix/cord-v2 (800 train / 100 validation / 100 test receipts). Labels were losslessly normalized before training.

Training ran well past convergence on purpose, so there was a wide range of adapters to pick from. Validation loss and the task metrics peak at different times here: loss starts rising around epoch 3, while the best field_f1 and nTED land at epoch 9 or later. The adapter published here is the one that scored best on the validation split by field_f1, and it was then scored once on the held-out test split. Early-stopping on loss would have picked a noticeably worse one.

Quantization

Quantized with GPTQModel (4-bit, group_size=128, symmetric, desc_act=false), which works out to roughly 4.29 bits per weight. Only the language-model decoder linears are quantized; the vision tower, the embeddings and lm_head stay in bf16.

Calibration used 256 multimodal chat samples from the CORD-v2 train split, each one a receipt image plus the fixed prompt plus the gold JSON answer. That makes this a data-aware quantization, and it shows: against the bf16 model it loses 0.0022 field_f1 and actually gains 0.0038 nTED, which is inside the noise of a 100-receipt split. A data-free bitsandbytes NF4 quantization of the same model came out measurably worse on both metrics.

Inference runs through the Marlin 4-bit kernel.

Limitations

• The test split is 100 receipts. Gaps of a point or two in field_f1 are inside the noise band and should be read as ties.
• CORD-v2 is photographed Indonesian retail and restaurant receipts. Expect worse results on other layouts, languages, or document types (invoices, handwriting).
• Field names mirror the raw CORD keys (nm, cnt, unitprice), not friendly names.
• Values are transcribed text, not validated arithmetic. Nothing checks that the line items sum to the total. Do not use the output for financial decisions without review.
• Schema-constrained decoding was tried and scored far worse than plain decoding (field_f1 falls roughly 35 points), so plain greedy decoding is what these numbers use and what is recommended.
• Any timing or throughput figure depends heavily on hardware. Measure on the machine you intend to deploy on.

License

apache-2.0, inherited from the base model. The CORD-v2 dataset carries its own terms; see its dataset card.