suture-8b

suture-8b

An 8B vision-language adapter that lists what changed between an underwriting binder and the issued policy.

Two page images in. Structured JSON out (PASS/FLAG + a 13-class discrepancy list). The oracle is a program, not a judge model.

Result

Seed-777, n=1000, 739 FLAG, zero signature overlap with train. Eval re-renders pages from structured fields.

The written contract is >0.95 recall, >0.95 precision, 1.00 HIGH. Rec and prec hold. HIGH is 0.969. DEDUCTIBLE_HIGHER 0.957 / 0.846 (was 0.507 / 0.407). Citation exact-match ≈ 1.0 is gold-string cloning; type recall is the money metric. Full tables in the repo docs/BENCHMARK.md.

Why this task is solvable

Issuance QC is a closed diff over a finite ISO/ACORD-style ontology (named insured, period, coverages, limits, deductibles, form numbers, additional insureds, premium). Carriers still run it with people and Guidewire / Duck Creek rules. There is no public VL benchmark for binder-vs-policy consistency. Nearby datasets (FUNSD, CORD, DocVQA, VRDU) extract fields; they do not ask whether two documents agree. Claims NLP is a different problem (fraud, coverage opinion) and needs a different oracle.

A seeded generator plus a deterministic verifier (suture_forge.verify) produces exact gold. The same recipe is what let a 1.7B LoRA beat a frontier model on ReconForge, a different closed task. Zero-shot general VL does not know this taxonomy — that is the 0.098 / 0.373 rows above.

How to load

from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

base_id = "Qwen/Qwen3-VL-8B-Instruct"
model = Qwen3VLForConditionalGeneration.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(model, "caiotheodoro/suture-8b")
processor = AutoProcessor.from_pretrained(base_id)

Two images: binder page, then policy page. Output schema is in the repo CONTRACTS.md. Training and eval in this project used Modal 4-bit, 384px (cloud/modal_eval.py).

Sample pages (forge-rendered, not carrier scans) under samples/: PASS pair, DEDUCTIBLE_HIGHER pair, AI_MISSING pair.

Training

Synthetic only. Seeds: 7 (train/val), 11, 13, 17, 19, 23 (class-conditional mixes). Seed 777 is eval-only. JSONL is on the dataset repo, not in git.

Chain: sft-vl → sft-distill → sft-restem → sft-numeric → sft-limithi → sft-ded → sft-dedhi. Continue-from-adapter SFT, 1 epoch per mix, Modal L4. This adapter: 1280 steps from sft-ded. This Hub repo is PEFT only (~200MB). No merged 16-bit 8B, no train JSONL here.

Limitations

• Forge-rendered pages, not carrier PDFs. Transfer is unmeasured.
• HIGH recall 0.969, not 1.00. Residual misses: AI_MISSING 0.912, FORM_MISSING 0.944.
• Research / reproduction of this benchmark. A human signs off on a live issuance file.

Citation

@software{theodoro2026suture,
  author = {Theodoro, Caio},
  title  = {Suture: vision-language policy-issuance QC with a verifier-as-oracle},
  year   = {2026},
  url    = {https://github.com/caiotheodoro/suture},
  note   = {seed-777 n=1000: recall 0.959, HIGH 0.969, precision 0.956, parse 1.0}
}