lightonocr-w9-finetuned

LightOnOCR-2-1B — W-9 Form Extraction

Fine-tuned version of LightOnOCR-2-1B-base for structured field extraction from IRS W-9 tax forms.

Model Description

• Base model: lightonai/LightOnOCR-2-1B-base
• Fine-tuning method: LoRA (r=16, alpha=32) via PEFT
• Training hardware: NVIDIA RTX 4000 Ada (24GB)
• Precision: bfloat16
• Output format: Minified JSON (not Markdown)
• Organization: Span Technology Services

Training Details

Evaluation Results

Evaluated on 61 held-out synthetic W-9 forms.

Per-Field Accuracy

Dataset

Trained on 500 synthetic W-9 forms generated programmatically:

• Official IRS W-9 PDF filled via PyMuPDF
• Realistic fake data via Faker (names, addresses, TINs)
• Rendered to PNG at 200 DPI
• Ground truth JSON derived directly from fill record (no re-OCR)
• No real PII used anywhere in training

Intended Use

• Extracting structured fields from IRS W-9 forms
• Self-hosted tax form processing pipelines
• Drop-in replacement for cloud VLM APIs in document extraction

Limitations

• Trained on synthetic data only — real-world scan quality may vary
• Optional fields (line_7_account_numbers, requester_name_address) have lower accuracy due to sparse training examples
• Designed for W-9 forms only — not tested on other form types

How to Use

import torch
from transformers import LightOnOcrProcessor, LightOnOcrForConditionalGeneration
from PIL import Image

model_id = "Aadhi852/lightonocr-w9-finetuned"

processor = LightOnOcrProcessor.from_pretrained(model_id)
model = LightOnOcrForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

image = Image.open("w9_form.png").convert("RGB")
prompt = "Extract all fields from this form and return a valid JSON object only. Do not include any explanation or additional text."

messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": prompt},
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
encoding = processor(images=image, text=text, return_tensors="pt")
encoding = {k: v.to(model.device) for k, v in encoding.items() if hasattr(v, 'to')}

with torch.no_grad():
    output_ids = model.generate(**encoding, max_new_tokens=512, do_sample=False)

input_len = encoding['input_ids'].shape[1]
result = processor.decode(output_ids[0][input_len:], skip_special_tokens=True)
print(result)  # {"name": "...", "tin": "...", ...}

Citation

If you use this model, please cite the base model:

• LightOnOCR-2-1B by LightOn