finetune-donut-cord-v3

Donut Base Fine-tuned on CORD v2 Dataset

Model Description

This model is a fine-tuned version of naver-clova-ix/donut-base on the CORD v2 (Consolidated Receipt Dataset) for OCR-free receipt understanding and structured information extraction.

Donut (Document Understanding Transformer) is an end-to-end vision-language model that processes document images directly to extract structured information without requiring an intermediate OCR step. This approach eliminates error propagation from OCR mistakes and provides a more streamlined solution for document understanding tasks.

Model Details

• Base Model: naver-clova-ix/donut-base (201M parameters)
• Task: Image-to-text generation for structured receipt information extraction
• Training Approach: Full fine-tuning (all 201M parameters updated)
• Hardware: RTX 5090 GPU with 32GB VRAM
• Training Framework: PyTorch Lightning with mixed precision (BF16)
• Dataset: CORD v2 - 800 training images, 100 validation images
• Image Resolution: 1280×960 pixels
• Max Sequence Length: 768 tokens

Performance

Test Set Results (100 held-out receipt images)

• Mean Accuracy: 90.73%
• Accuracy Range: 27% - 100% across samples
• Performance by Complexity:
  • Simple receipts (3-5 items): 95-100% accuracy
  • Complex receipts (15+ items): 75-80% accuracy

Evaluation Metric

Accuracy is calculated using the JSONParseEvaluator from the Donut framework, which performs field-level comparison of extracted information:

Accuracy = (Number of Correctly Extracted Fields) / (Total Number of Fields)

The evaluator compares predicted and ground truth JSON objects by matching key-value pairs across multiple fields including:

• Menu items (name, count, unit price)
• Subtotals
• Tax amounts
• Service charges
• Total amounts

Note: The 90.73% accuracy means that on average, approximately 9 out of every 10 information fields are correctly extracted from unseen receipts.

Training Details

Hyperparameters

Training Configuration

• Validation Frequency: Every 0.2 epochs for fine-grained monitoring
• Model Checkpointing: Based on validation accuracy
• Optimization Techniques:
  • Linear warmup scheduling
  • Gradient clipping for stability
  • Early stopping to prevent overfitting

Resource Utilization

• Peak GPU Memory: ~9GB (28% of available 32GB)
• Training Time: ~45 minutes per epoch
• Power Consumption: 40-75W during training

Usage

Installation

pip install transformers torch pillow

Inference Example

import re
import json
import torch
from PIL import Image
from transformers import DonutProcessor, VisionEncoderDecoderModel

# Load model and processor
processor = DonutProcessor.from_pretrained("abdirisak874/finetune-donut-cord-v3")
model = VisionEncoderDecoderModel.from_pretrained("abdirisak874/finetune-donut-cord-v3")

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

# Load and preprocess image
image = Image.open("path/to/receipt.jpg").convert("RGB")
pixel_values = processor(image, return_tensors="pt").pixel_values
pixel_values = pixel_values.to(device)

# Prepare decoder input
task_prompt = "<s_cord-v2>"
decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids
decoder_input_ids = decoder_input_ids.to(device)

# Generate prediction
outputs = model.generate(
    pixel_values,
    decoder_input_ids=decoder_input_ids,
    max_length=model.decoder.config.max_position_embeddings,
    early_stopping=True,
    pad_token_id=processor.tokenizer.pad_token_id,
    eos_token_id=processor.tokenizer.eos_token_id,
    use_cache=True,
    num_beams=1,
    bad_words_ids=[[processor.tokenizer.unk_token_id]],
    return_dict_in_generate=True,
)

# Decode output
sequence = processor.batch_decode(outputs.sequences)[0]
sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
result = processor.token2json(sequence)

print(json.dumps(result, indent=2))

Expected Output Format

{
  "menu": [
    {
      "nm": "ITEM NAME",
      "cnt": "1",
      "price": "10.00"
    }
  ],
  "sub_total": {
    "subtotal_price": "10.00"
  },
  "total": {
    "total_price": "11.00",
    "tax_price": "1.00"
  }
}

Limitations and Known Issues

Current Limitations

1. Complex Layout Handling: Performance degrades on receipts with 15+ line items or dense nested structures
2. Long Item Names: Tendency to truncate very long menu item names
3. Character Substitution: Occasional OCR-like errors (e.g., 'O' vs '0', 'l' vs '1')
4. Price Formatting: Some inconsistencies in decimal point placement
5. Language Support: Primarily trained on Indonesian restaurant receipts; may require fine-tuning for other languages or receipt formats

Error Distribution

Recommended Use Cases

✅ Good for:

• Indonesian restaurant receipts
• Receipts with 3-15 line items
• Standard receipt layouts
• Batch processing of similar receipt formats

⚠️ Use with caution:

• Receipts with 15+ items
• Non-standard layouts
• Heavily damaged or low-quality images
• Multi-lingual receipts

Training vs. LoRA Comparison

This model was trained using full fine-tuning (updating all 201M parameters) rather than parameter-efficient methods like LoRA. The comparison:

Full fine-tuning proved essential for document understanding tasks requiring complex visual-linguistic alignment, despite higher computational costs.

Research and Citation

This model was developed as part of research on OCR-free document understanding. For detailed methodology, training procedures, and analysis, please refer to the accompanying research paper.

Key Findings

1. Document understanding requires extensive parameter updates across both vision and language components
2. Parameter-efficient methods like LoRA are insufficient for complex multimodal tasks
3. Proper optimization techniques (warmup, gradient clipping, early stopping) are critical for stability
4. Mixed precision training (BF16) enables practical implementation with reasonable memory requirements

Future Improvements

Potential areas for enhancement:

• Knowledge distillation to create smaller, faster models
• Data augmentation for improved robustness
• Extension to other document types (invoices, forms, IDs)
• Multi-language support
• Ensemble methods for challenging cases

Contact and Feedback

For questions, issues, or suggestions for improvement, please open an issue on the model repository or contact the model author.

License

This model is released under the OpenRAIL license, inheriting from the base Donut model license.

Acknowledgments

• Base model: naver-clova-ix/donut-base
• Dataset: CORD v2
• Training infrastructure: vast.ai GPU rental services