layout-aware-receipt-ocr

Layout-Aware Receipt Information Extraction

This repository contains the trained PyTorch weights for an end-to-end Visually Rich Document Understanding (VRDU) pipeline. The system is custom-built to extract key entities from receipts (based on the ICDAR 2019 SROIE dataset) by simultaneously analyzing text semantics and physical document layout.

Neural Network Architecture

• OCR Engine (CRNN): A Convolutional Recurrent Neural Network (CNN + 2-layer Bi-LSTM) trained with CTC Loss to accurately decode raw text from cropped image regions.
• Spatial Intelligence (GNN): A Multi-Head Graph Attention Network (GAT) featuring 128 hidden channels. It processes dynamic graphs where each node is a 392-D multi-modal vector (combining 384-D Sentence-Transformer semantic embeddings, 4-D spatial coordinates, and 4-D custom financial/date heuristics).

Quantitative Evaluation

The pipeline was rigorously evaluated on the unseen SROIE 2019 test split.

1. OCR Engine (CRNN) Performance

Evaluated over 33,618 cropped word instances to measure text extraction exactness.

2. Layout-Aware GNN Performance

Evaluated over 4,835 test sentences, prioritizing the F1-Score to counteract the severe class imbalance inherent in receipt data (where background noise vastly outweighs key entities).

Intended Use

These weights (.pth files) are designed to be loaded into the custom PyTorch inference pipeline. For the complete training codebase, data preprocessing modules, and end-to-end inference scripts, please refer to the associated GitHub repository.