qwen3-vl-8b-constat-amiable-lora

Qwen3-VL-8B — Constat Amiable Extractor (LoRA adapter)

A LoRA fine-tune of Qwen3-VL-8B-Instruct that reads a photo or scan of a French constat amiable d'accident automobile (the European Accident Statement) and returns a single JSON object with a fixed 118-field schema — handwritten text, printed text, and checkbox states.

[!IMPORTANT] This model was trained and evaluated entirely on synthetic data. No real accident reports were used at any stage. The evaluation below measures performance on the synthetic distribution the model was trained on — it is not a measurement of real-world accuracy. Please read Limitations & honest caveats before using or citing these numbers.

---

TL;DR

The point of this project is to show that a small, cheap, fully-synthetic LoRA fine-tune can specialize an open 8B vision-language model on a narrow document-extraction task. It is a methodology demonstration and a portfolio artifact — not a claim of production-grade or real-world state-of-the-art accuracy.

---

Table of contents

1. What the model does
2. Output schema
3. How to use
4. Training data
5. Training procedure
6. Evaluation
7. Limitations & honest caveats
8. License & legal
9. Benchmark scripts & transparency

---

What the model does

The constat amiable is the standardized European Accident Statement that two drivers fill in by hand after a road accident. It is a dense, two-column form: identity and address of each party, vehicle and insurance details, a 17-item list of accident circumstances expressed as checkboxes per vehicle, free-text observations, and a sketch ("croquis") of the scene.

Given one image of such a form, the model produces one JSON object:

• text fields — transcribed verbatim (names, addresses, plate numbers, dates, policy numbers, free-text observations…), or null when the field is blank or illegible;
• checkbox fields — true / false for every checkbox, including the 17 circumstance boxes for each vehicle.

The model is driven by a fixed system prompt (a "forensic document examiner" role with anti-hallucination, checkbox and JSON-formatting protocols) plus a short user instruction. Both are shipped in prompts/ and must be used at inference time — the model was trained with them.

Intended use

• Research and demonstration of synthetic-data specialization for VLMs.
• Extracting fields from constat images of the template family seen in training, in a setting where outputs are human-reviewed.

Out of scope

• Production insurance processing without a prior validation campaign on real, labelled constats.
• Any automated legal, financial or claims decision.
• Documents outside the trained template layouts, other form types, or other languages.
• Treating the JSON output as ground truth — VLM outputs must be verified.

---

Output schema

The schema has 118 keys in a flat object (no nesting). Breakdown:

• 73 text fields → string or null
• 45 checkbox fields → boolean (true / false); croquis_ is true / null

Logical groups (keys suffixed _A / _B for vehicle A / vehicle B):

The complete, authoritative schema (with the per-field type annotations the model was trained against) is in prompts/new_system_prompt.md.

---

How to use

The released artifact is a LoRA adapter. Load the base model, apply the adapter, and prompt with the shipped system prompt + instruction.

import json
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel

BASE = "Qwen/Qwen3-VL-8B-Instruct"
ADAPTER = "hasanbasbunar/qwen3-vl-8b-constat-amiable-lora"          # this adapter

processor = AutoProcessor.from_pretrained(BASE)
model = Qwen3VLForConditionalGeneration.from_pretrained(BASE, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# Prompts shipped with the model (prompts/ folder)
SYSTEM_PROMPT = open("prompts/new_system_prompt.md", encoding="utf-8").read()
INSTRUCTION   = open("prompts/instruction.txt", encoding="utf-8").read().strip()

image = Image.open("constat.jpg").convert("RGB")
messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
    {"role": "user", "content": [
        {"type": "text",  "text": INSTRUCTION},
        {"type": "image", "image": image},
    ]},
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=3072, do_sample=False)  # greedy
text = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
data = json.loads(text)

Notes:

• The model was trained to emit raw JSON (no markdown fence, no preamble).
• Greedy decoding (do_sample=False) is recommended for a deterministic, reproducible extraction.
• Training and the benchmark used the Unsloth 4-bit build (unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit); 4-bit inference reproduces the reported numbers and fits comfortably in <24 GB.

---

Training data

100 % synthetic, procedurally generated. No real constat was collected, scanned or used. The generator (main.py) builds each sample from:

• a small set of blank constat amiable form layouts — the standardized European Accident Statement — used as the backgrounds onto which the synthetic fields are rendered;
• realistic fake content from Faker, locale-weighted ~85 % fr_FR and ~3 % each of de_DE, en_GB, es_ES, it_IT, nl_NL, plus domain dictionaries for vehicles, insurers, European countries, postal-code formats, place names, etc.;
• handwriting simulation — 304 fonts (filtered for French-glyph coverage) to render the "handwritten" entries;
• photo-realistic degradation applied with calibrated probabilities, so the model sees the kind of images it would meet in the wild: fold lines, coffee stains, rain drops, shadows & highlights, blur spots, brightness/contrast variation, Gaussian blur, sensor noise, perspective distortion, rotation (±6°), and JPEG recompression (quality 15–100, weighted toward smartphone-grade 65–90);
• final images resized to 1536–2240 px and aligned to the 32-pixel grid Qwen3-VL requires.

Each sample ships with a perfect, machine-generated ground-truth JSON — the key advantage of synthetic data: exhaustive, noise-free labels at scale, with no privacy exposure (all identities are fake).

The held-out evaluation set (599 samples) was produced by a separate generation run with the same generator — see Evaluation.

The honest trade-off: synthetic data is scalable and perfectly labelled, but it only approximates real handwriting, real paper wear and real photo conditions. The gap to real-world constats is real and is not measured here.

---

Training procedure

QLoRA fine-tuning with Unsloth FastVisionModel + TRL SFTTrainer. LoRA adapters were placed on both the vision and the language layers (attention + MLP).

Loss. The first training steps logged a loss around 0.17; mean training loss over the run was 0.0798, and validation loss (128 held-out samples) reached 0.0535 at the end of training.

Note: this is a deliberately light fine-tune — 200 steps is roughly a quarter of one epoch. The validation-loss curve was still trending down, so the figures below should be read as a lower bound on what this recipe can reach with more training. Cross-entropy on a highly templated JSON target is also a weak proxy for extraction quality — the real measure is the field-accuracy benchmark below.

---

Evaluation

Methodology

The goal is an honest, transparent, auditable comparison. The benchmark scripts are published in full (see Benchmark scripts & transparency); the synthetic evaluation set itself is not released.

Held-out set. A separate run of the same generator produced 599 fresh synthetic constats (holdout_eval). Because every sample is an independent random draw, these constats do not appear in the training set — they are a held-out sample of the training distribution. The benchmark uses the first 64 of them.

Models compared (same 64 images, same prompts):

Each model is evaluated in its realistic deployment mode. Gemini is given structured output — a JSON schema it must conform to — because that is how one would actually deploy it for extraction; this is the configuration that gives Gemini its best, most reliable result (it is not a handicap). The fine-tuned model uses free-form generation because it was trained to emit the exact schema and does so reliably without constraint. All three receive the identical system prompt and instruction.

Gemini settings. temperature and thinking_level left at their defaults — Google explicitly advises against lowering the temperature for Gemini 3, as it can cause looping/degradation. max_output_tokens = 32768. All 64 requests succeeded on the first call (0 retries, 0 API failures); the retry logic in the script triggers only on transport/HTTP errors, never on output quality.

Scoring. All three models are scored with the same metric definitions, gathered in the module eval_metrics.py. Comparison is case-insensitive and whitespace-normalized (a line break inside a form field is not a content difference). With only 64 samples, the dominant source of uncertainty is sampling noise, not the metric — see Limitations.

Metrics

Why two accuracy numbers? A real constat leaves many fields blank, and "blank ↔ blank" counts as a correct match. So field_acc is optimistic — it is partly rewarded for agreeing on empty fields. filled_acc is the number that matters: accuracy on the content actually present on the form.

Results

Benchmark on n = 64 held-out synthetic constats.

Chart: CER is shown as character accuracy (1 − CER) so every bar points the same way (higher = better); exact_full is omitted — it is ≈ 0 % for all three models. Full numbers in the table.

exact_full is near zero for every model: it scores 1.0 only when all 118 fields are correct at once, so a single slip anywhere drops the sample to 0. It is an intentionally brutal metric kept only for completeness — field_acc, filled_acc and cer are the informative ones.

Interpretation

Fine-tuning delivers a large, consistent gain over the zero-shot base — on every metric:

This is the result the project set out to demonstrate: a light, fully-synthetic LoRA fine-tune (200 steps) strongly specializes an open 8B vision-language model for this extraction task.

On this benchmark the fine-tuned model matches or exceeds Gemini 3.1 Pro on every metric (tied at 100 % JSON validity, ahead on all the others) — but that comparison must be read carefully:

[!WARNING] This is not a symmetric comparison. The fine-tuned model was trained on this exact synthetic style and is evaluated on a held-out sample of the same distribution. Gemini 3.1 Pro and the base model see this style for the first time. The fine-tuned model's lead therefore measures how well fine-tuning specialized it to the target distribution — it does not show that the fine-tuned model is better than Gemini at reading real constats. Gemini's score is best read as "what a strong general-purpose model achieves zero-shot on this layout"; the fine-tuned score as "what task-specific specialization achieves on its own distribution." A fair real-world comparison would need a labelled set of real constats, which this project does not have (see Limitations).

Honest takeaway. For a narrow, well-defined document-extraction task, fine-tuning a small open model on synthetic data is remarkably effective: a 200-step LoRA was enough to take an 8B model from clearly behind to — on this in-distribution benchmark — ahead of a frontier model, at a fraction of the size and with no per-call API cost. How much of that advantage survives on real-world documents is unmeasured, and is the main open question this project does not answer.

filled_acc (76.6 %) is the most informative single number: of the fields that actually carry content, roughly three in four are extracted exactly right — strong for a 200-step fine-tune, with clear headroom remaining.

---

Limitations & honest caveats

Read this section before trusting any number above.

1. In-distribution evaluation — the comparison is not symmetric. The fine-tuned model was trained on this exact synthetic style and is then tested on a held-out sample of the same distribution. The base model and Gemini 3.1 Pro see this style cold. A higher score for the fine-tuned model therefore demonstrates successful specialization to the target distribution — it does not establish that the fine-tuned model is better than Gemini at reading real constats.

2. Synthetic ≠ real. Training and evaluation are 100 % synthetic. Real constats have real handwriting, real paper wear, and real photo conditions that the generator only approximates. Real-world accuracy is unmeasured.

3. No real labelled test set. The project has no corpus of real, annotated constats, so no real-world claim can be made. Anyone deploying this model should first run a validation campaign on real data.

4. Small evaluation set (n = 64). Point estimates carry a sampling uncertainty on the order of several percentage points. Treat differences of a few points as noise.

5. Light fine-tune. 200 steps ≈ 0.23 epoch; the model is under-trained relative to the 14k samples available. The reported numbers are a lower bound for this recipe, not its ceiling.

6. Template coverage. Only the ~5 template layouts seen in training are supported. Other constat layouts, other insurers' forms, or other document types are out of scope.

7. Hallucination. Like any VLM, the model can produce plausible-looking but wrong field values, especially on illegible handwriting. The system prompt instructs it to return null when unsure; this mitigates but does not eliminate the risk. Outputs must be human-reviewed.

8. Privacy / GDPR. Training identities are fake (Faker). But real constats contain personal data — names, addresses, plate numbers, dates of birth. Any real-world use must comply with GDPR and applicable insurance-data regulations.

---

License & legal

• Adapter weights — released under Apache-2.0, inherited from the base model Qwen/Qwen3-VL-8B-Instruct (Apache-2.0). The adapter contains only low-rank weight deltas; it does not embed any template image or any training sample.
• Base model — you must obtain Qwen3-VL-8B-Instruct separately, under its own Apache-2.0 license.
• Datasets. The synthetic training and evaluation sets are not released. This card describes how they were generated; the data itself is kept private.
• Form. The constat amiable is the standardized European Accident Statement — a public administrative form that insurers distribute freely to every driver. The generator uses blank layouts of this standard form purely as rendering backgrounds; the released adapter contains only weight deltas and does not embed or reproduce any form image.

---

Benchmark scripts & transparency

All three benchmark scripts are published in full so the exact scoring methodology can be read and audited:

Both scripts also write a per-sample *_details.jsonl — the raw model prediction and the ground truth for every sample — used to inspect and verify results.

[!NOTE] The synthetic evaluation set itself is not released (see License & legal). The scripts are published for methodological transparency — so the scoring logic can be independently checked — not as a turnkey reproduction package.

How the runs were launched:

# Base vs fine-tuned (same 64 hold-out images, greedy decoding)
EVAL_N=64 EVAL_ADAPTER_PATH=./lora_model_XXXX \
  python eval_zeroshot_vs_finetuned.py

# Gemini 3.1 Pro (same 64 images, structured output)
EVAL_N=64 GEMINI_API_KEY=... \
  python eval_gemini.py

EVAL_N must be identical across the two scripts for the comparison to be valid.

---

Acknowledgements

• Qwen team / Alibaba — Qwen3-VL-8B-Instruct.
• Unsloth — efficient QLoRA training.
• Trained on an NVIDIA DGX Spark.

Model card last updated: 2026-05-18.