tiny-receipt-vqa-structured-qa-21m

TinyReceiptVQA-StructuredQA-22M-BPE1536-DirectNoValue-LoRA-Router-e100

TinyReceiptVQA is a compact receipt-domain visual question-answering model. It takes a full receipt image and a question, then generates a structured rationale and final <answer> value. A learned router selects one of eight family slots; each slot has a post-encoder memory adapter and a post-decoder adapter for phone, address, store, item-row, item-math, item-lookup, math, or other questions.

Files

encoder_model.onnx       # encoder + router + memory adapter + cross K/V projection
decoder_model.onnx       # one-token FP32 decoder with self K/V cache
encoder_model_int8.onnx  # static W8A8 encoder with the same decoder-state contract
decoder_model_int8.onnx  # static W8A8 one-token decoder
manifest.json            # graph contract, hashes, and runtime validation
config.json
vocab.json
byte_bpe.py              # dependency-free tokenizer runtime
inference.py             # ONNX Runtime greedy decoding
examples/
  receipt_en.jpg         # verified English synthetic receipt
  receipt_ko.jpg         # verified Korean synthetic receipt
  family_examples.json   # 16 explicit/AUTO checks and expected answers
  test_all_families.py   # emits decoded text, token IDs, and BPE tokens
eval/
  heldout_summary.json   # heldout evaluation summary
  int8_w8a8_summary.json
  fp32_int8_heldout_summary.json
  runtime_benchmark_fp32_cpu.json
  runtime_benchmark_int8_cpu.json
  comparison.json         # compact current evaluation/runtime summary

The FP32 and INT8 models use KV caching. The encoder runs once and emits per-layer cross-attention K/V. The decoder then consumes one token at a time while carrying its self-attention K/V and target-padding-mask outputs. These four ONNX graphs are the complete distributed model set.

Usage

pip install -r requirements.txt

python inference.py \
  --model-dir . \
  --image examples/receipt_en.jpg \
  --question "What is the street number in the store address?"
# Expected JSON fields: "answer": "837", "well_formed": true

# Use the Korean example with the static W8A8 U8S8 QDQ ONNX files.
python inference.py \
  --model-dir . \
  --precision int8 \
  --image examples/receipt_ko.jpg \
  --question "영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까?"
# Expected JSON fields: "answer": "875", "well_formed": true

--family auto is the default and uses the learned router. An explicit family such as phone or address can be supplied for controlled debugging.

The input image contract is float32 [batch, 1, 320, 672]: convert to grayscale, resize to 672x320 (width x height), scale to [0,1], then normalize with (x - 0.5) / 0.5.

Runtime interface

The opset is 18. Batch, question length, memory length, and past/present cache lengths are dynamic; image height and width are fixed.

• Encoder inputs: image, question_ids, and family_ids (-1 means AUTO).
• Encoder outputs: memory, padding mask, router logits, selected family, and cross_k_0/v_0 through cross_k_3/v_3.
• Decoder inputs: one decoder_input_ids token, per-sample position_ids, selected family, encoder/target padding masks, per-layer Cross K/V, and per-layer prior Self K/V.
• Decoder outputs: one-token logits, updated target padding mask, and per-layer present Self K/V. External cache tensors remain float32 in the W8A8 variant.

Use the included inference.py entry point to handle preprocessing, AUTO-routing, and greedy generation.

The JSON result sets well_formed to true only when generation contains a complete <answer>...</answer> block. If that block is missing, answer is an empty string and well_formed is false; inspect generated for diagnostics.

Byte-Fallback BPE and vocabulary

This release uses only the packaged Byte-Fallback BPE1536 tokenizer contract. vocab.json declares type=byte_fallback_bpe, version 1, NFC normalization, and exactly 1,536 tokens. Character-vocabulary artifacts are rejected instead of falling back silently. Its SHA-256 tokenizer fingerprint is 5801826ac9092bdc984210af805d2ff20f0398b80adaf5963ef13b04771a8584.

The structured tags and digits never participate in merges. Whitespace is a hard merge boundary; in this vocabulary an ASCII space is <0x20> (ID 54). If a Unicode character is absent from the learned alphabet, its UTF-8 bytes are emitted instead, so valid Unicode input can round-trip without using <unk>: decode(encode(text)) == NFC(text). Byte fallback guarantees representability, not that the vision model can recognize an unseen character correctly.

Only training records were used to learn the alphabet and merges. The optional tokenizers library was used while building the vocabulary, but packaged loading, encoding, and decoding use the dependency-free byte_bpe.py reference runtime. Browser implementations must reproduce its NFC, whitespace-boundary, atomic-token, merge-order, and byte-fallback behavior exactly.

The target schema is direct_answer_no_value_v1. For direct reads such as a phone number, store name, or item name, <value> would merely duplicate the answer and is omitted. It remains for address extraction, lookup, and arithmetic when it carries distinct evidence.

Training used BF16 autocast on the GPU. Saved source tensors and the FP32 ONNX graphs are FP32; this is expected and does not mean training was performed in FP32. The additional deployment graphs are static W8A8 INT8.

Verified bilingual examples

These receipts come from the synthetic validation split, not heldout data or INT8 calibration data. They were selected as readable demonstrations and are not a family-level benchmark.

All 16 questions passed in every tested combination:

AUTO routes the two listed math questions to trained item_math; their final answers and decoded outputs remain exact. The first six listed families have training records. math and other have zero family-specific training records, so the explicit math rows are execution smoke tests, not evidence of general math-family accuracy. other is not included.

Observed decoded output by family

The following outputs were observed identically for FP32 and INT8, with both explicit and AUTO routing, on the English example. These are complete decoder outputs after removing <bos>/<eos> for display.

The Korean outputs use the same schema:

Actual emitted BPE tokens

These are the actual FP32 decoder tokens for the seven English family rows, not strings decoded and then re-encoded. The selected examples emitted the same token sequences in INT8.

For the English phone-position question, the observed decoded output was:

<field>phone</field><value>6533831</value><op>front_2</op><answer>5</answer>

Its actual token IDs and tokens were:

IDs:    [1, 4, 1038, 5, 6, 18, 17, 15, 15, 20, 15, 13, 7,
         8, 1063, 14, 9, 10, 17, 11, 2]
Tokens: [<bos>, <field>, phone, </field>, <value>, 6, 5, 3, 3, 8, 3, 1,
         </value>, <op>, front_, 2, </op>, <answer>, 5, </answer>, <eos>]

The Korean back-position question generated <field>phone</field><value>9113125</value><op>back_3</op><answer>1</answer>.

Run the packaged test from the repository root. It reports decoded text, actual decoder IDs, token strings, selected families, and router logits:

python examples/test_all_families.py \
  --model-dir . \
  --precision both \
  --routing both

# Learned router only (family_ids=-1).
python examples/test_all_families.py --model-dir . --routing auto

The machine-readable questions and recorded answers are in examples/family_examples.json.

The INT8 variant uses static U8S8 W8A8 QDQ quantization for convolution, attention, FFN, adapter, router, and output-head Conv, MatMul, and Gemm operators. Weights are symmetric signed INT8 per output channel, activations are asymmetric UINT8 per tensor, and integer accumulation is INT32. Calibration uses only synthetic training records; heldout records are not calibration inputs. ONNX Runtime can fuse supported QDQ regions into integer kernels, while normalization, softmax, positional, and shape operations remain floating point. The two FP32 graphs total 86.2 MiB; the matching W8A8 graphs total 27.4 MiB, a 68.2% reduction. These are executable quantized ONNX graphs, not storage-only weights that are dequantized back to FP32 before inference.

Validation

All four model graphs pass ONNX Checker and CPU ONNX Runtime execution. Validation covered AUTO routing, all eight explicit adapters, dynamic sequence shapes, and greedy generation. The selected adapter ID matched exactly.

On the 2,000-question AUTO phone/address heldout evaluation, the model achieved answer exact 97.25% and target exact 47.35% (FP32); the static W8A8 graph achieves 97.10% and 47.20%.

Address target exact is low because the target includes a full transcription of the address text in <value>, which this model rarely reproduces character for character. The <answer> it is asked for — the street number — is correct 95.35% of the time.

The source PyTorch reference and the final ONNX FP32 rerun agree exactly on all nine reported accuracy metrics across the overall, address, and phone subsets.

Character error rate

CER is the standard OCR edit-distance metric:

CER = (substitutions + deletions + insertions) / ground-truth characters.

For readability, the table reports character accuracy, 1 - CER, alongside CER. These are NFC-normalized PyTorch FP32 AUTO results for the same 2,000 heldout examples, computed from the FP32 ONNX rerun. Lower CER and higher character accuracy are better.

Full-target character metrics include repeated structured markup such as <field>, <op>, and <answer>. Those common tag characters are easy to generate and therefore make full-target character accuracy look better than the model's actual OCR/copy quality. Use <value> CER to assess transcription of the visual evidence and <answer> exact/CER to assess final QA output.

The final default KV-cache ONNX files were also evaluated directly on all 2,000 heldout questions with ONNX Runtime 1.23.2 and CPUExecutionProvider, using learned AUTO routing:

The two variants had 74.45% full prediction agreement, 97.95% answer agreement, and 100.00% selected-family agreement. There were 511 changed full predictions out of 2,000. The INT8 answer-exact result was 0.25 percentage points below FP32. Variant-specific accuracy is reported independently above.

With batch size 24 and 12 CPU threads, accumulated model execution time for all records was 273.27 seconds for FP32 and 257.15 seconds for W8A8. These aggregate evaluation timings are not single-query latency measurements; the one-thread per-VQA runtime measurements are reported below. See eval/fp32_int8_heldout_summary.json for the complete summary.

Model design and architecture

This configuration has 21,834,104 train-time parameters (about 21.8M, rounded to 22M in the model name), and all of them were trainable. The token embedding and output projection share one weight matrix, so it is counted once. The LoRA branches account for 256,000 parameters during training; they are folded into dense FFN weights for ONNX inference and add no runtime branch.

672x320 grayscale receipt -> stride-32 CNN -> 10x21 visual tokens -----------+
question -> Byte-Fallback BPE -> question embeddings -----------------------+--> concat
question embeddings -> mean-pool router -> selected family -----------------+      |
                                                                                   v
                              6-layer multimodal encoder -> selected memory adapter
                                                                                   |
                                                               memory mask + per-layer cross K/V
                                                                                   |
BOS / current token + prior self K/V -> 4-layer incremental decoder --------------+
                      -> selected decoder adapter -> tied vocabulary head -> next token

Vision and multimodal encoder

The input is float32 grayscale [B, 1, 320, 672]. Five bias-free 3x3 convolutions downsample through 1 -> 48 -> 96 -> 192 -> 320 -> 320 channels. Conv blocks use GroupNorm and SiLU; four two-convolution residual blocks are interleaved with the downsampling stages. The total stride is 32, producing a 10x21 feature map that is flattened into 210 visual tokens of width 320.

Visual tokens receive learned image positions and an image-type embedding. Question tokens use the shared 1536x320 token embedding, learned question positions, and a question-type embedding. The model concatenates [visual tokens, question tokens] and applies a 6-layer pre-LayerNorm Transformer encoder with 8 attention heads (head width 40), FFN width 1280, GELU, and dropout 0.1. The encoded memory shape is [B, 210 + Q, 320].

Router and family adapters

The router sees question embeddings only, before image/question fusion. It mean-pools non-padding question tokens, then applies Linear(320 -> 160), GELU, and Linear(160 -> 8). AUTO routing takes the hard argmax; family_ids=0..7 explicitly overrides it. The ordered slots are phone, address, store, item_row, item_math, item_lookup, math, and other.

One selected residual bottleneck adapter is applied after the encoder and one after the decoder. Each is 320 -> 64 -> 320 with GELU, so this is eight routed family slots backed by 16 adapter MLPs, not a soft mixture of all adapters. Training used token cross-entropy plus 0.1 * router-family cross-entropy. Family-specific training records cover the first six slots. math and other are untrained extension slots whose encoder and decoder adapter up-projections are exactly zero. Explicitly selecting either slot therefore applies an identity adapter; AUTO can instead route such prompts to a trained family.

Decoder, LoRA, and deployment split

The 4-layer pre-LayerNorm Transformer decoder uses causal self-attention and cross-attention to multimodal memory, with the same 320-wide, 8-head, 1280-wide-FFN design. A final LayerNorm and output bias produce token logits. The output projection is weight-tied to the input token embedding.

Training added rank-8 LoRA (alpha=16, scale 2, dropout 0.05) to linear1 and linear2 in every encoder and decoder FFN: 20 Linear modules in total. This was LoRA-augmented full-model training, not a frozen-base PEFT run. Export folded all 20 deltas into their dense weights; the maximum pre/post-merge absolute difference was 5.245208740234375e-06. The ONNX graphs contain no separate LoRA operators.

The split encoder runs the CNN, multimodal encoder, router, memory adapter, and four per-layer cross-attention K/V projections once. The default FP32 and W8A8 decoders consume one token per call and carry headed self-attention K/V plus the target padding mask between calls. The encoder padding mask remains active in every cross-attention operation. On CPUExecutionProvider with one thread, median-of-per-VQA median model E2E latency was 288.50 ms for FP32 and 173.67 ms for W8A8. The reports include measurements for each of the 20 VQAs: eval/runtime_benchmark_fp32_cpu.json and eval/runtime_benchmark_int8_cpu.json. The 2,000-question accuracy comparison above is the quality check for both variants. Generation uses greedy argmax from BOS until EOS or 192 tokens.

The source checkpoint is epoch-100 last.pt. Heldout was excluded from training and intermediate checkpointing, then used post hoc to compare e79 best.pt with e100 last.pt; e100 last.pt was selected for higher heldout answer exact.

Target serialization: direct_answer_no_value_v1

This is the current training-target serialization identifier; it is not a model-architecture or tokenizer version. The complete serialized target is teacher-forced into the decoder and contributes to token cross-entropy loss:

<field>...</field>[optional <value>...</value>]<op>...</op><answer>...</answer>

<field>, <op>, and <answer> are always retained. <value> is omitted only when it would repeat the final answer: identity, copy, count, and item_N_name, item_N_unit_price, item_N_quantity, or item_N_total. It remains when it carries distinct evidence for extraction, lookup, or arithmetic. For example:

<field>store</field><op>identity</op><answer>ABC마트</answer>
<field>phone</field><value>6533831</value><op>front_2</op><answer>5</answer>

The separate answer property in an annotation is used for evaluation; it is not a second decoder-loss target. Using the v1 identifier does not alter the learned output behavior or model weights, so retraining is unnecessary.

Limitations

Training and task scope

This checkpoint learned a finite bilingual structured_qa task grammar for receipts. It is not an open-world document-VQA system, a general-purpose OCR engine, or an arbitrary table-query/calculation model. Training used synthetic receipts together with annotated real receipts, but this release does not ship the exact training-record count or per-family training distribution.

The prompt generator selects from finite English and Korean template banks; question language may differ from the receipt text language. Byte fallback makes other Unicode text encodable, but it does not add multilingual OCR or QA capability. Close human paraphrases are therefore not guaranteed. For example, prefer an explicit learned form such as 전화번호 앞에서 1번째 숫자는? or What is the first digit of the store phone number?. A shorter form such as 전화번호 첫번째 숫자, wording with index, or a request for the fifth digit is outside the guaranteed template/range and may produce a plausible but wrong operation.

Only phone, address, store, item_row, item_math, and item_lookup have family-specific training records. math and other are untrained extension slots with exact identity adapters; explicitly selecting either slot does not provide a separately trained math or catch-all capability. Direct extraction of dates, times, tax, payment method, printed subtotal/grand total, change, and other fields outside the table above has no task-specific accuracy claim.

Images are converted to grayscale and resized to the fixed 672x320 input. Training augmentation covered modest contrast/brightness changes, blur, and small rotations, so tiny text, heavy blur, large rotation, unusual layouts, or very dense receipts may degrade recognition. CER in this card is informational for transcription within the evaluated tasks; it is not evidence of unrestricted full-document OCR quality.

Output and evaluation scope

• well_formed: true means only that a complete <answer>...</answer> block was generated. It does not verify that the selected family, <op>, copied <value>, or answer is semantically correct.
• The 2,000-record heldout evaluation contains only supported phone/address prompts (1,032 address and 968 phone). It does not validate store/item families, every trained operation, or novel paraphrases. The bilingual family examples are selected smoke tests, not a family-level benchmark.
• Heldout was excluded from training and intermediate checkpointing. After training it was used to compare e79 best.pt and e100 last.pt and to select e100 last.pt, so it is a development test rather than a final blind benchmark.
• W8A8 output can differ from FP32 output because both weights and activations are quantized. Validate accuracy and latency in the target browser.
• Integer-kernel fusion depends on the ONNX Runtime execution provider and target hardware.