closing-doc-extraction-qwen3-8b-lora

closing-doc-extraction-qwen3-8b-lora

A LoRA adapter for mlx-community/Qwen3-8B-4bit that extracts a fixed JSON schema of key fields from U.S. (Florida) real-estate closing documents.

Read this first: what this adapter is and is not

This adapter was trained on a corpus that is overwhelmingly synthetic:

• 292 of 302 training documents are synthetic. They were generated locally by qwen3:32b from a small set of Florida closing-document templates (deeds, mortgages, closing disclosures, title commitments, condo/HOA disclosures, filled contracts) with fabricated names, parcels, book/page, and instrument numbers. Only 10 documents (real CFPB and FAR/BAR reference PDFs) are genuinely real.
• The training labels came from the same model family. Every assistant-side JSON was produced by qwen3:32b and validated for schema conformance. There is no independent human ground truth, so the adapter learns to reproduce qwen3:32b's extractions on templated text.
• The held-out test set is same-distribution synthetic data. Its near-perfect scores measure in-distribution template matching, not real-world extraction. They do not show that the model extracts correctly from real, messy, OCR'd closing documents it has never seen.

Treat this as a demonstration of the fine-tuning workflow and an in-distribution result, not a production-ready document extractor. The only genuinely out-of-distribution measurement here is the 10 real forms (below), and even those carry qwen3:32b-generated labels.

What it does

Given the raw text of a closing document in the user turn, the model returns a single JSON object with exactly these keys (null where a field is absent):

{
  "document_type": null,
  "parties": {"grantor": null, "grantee": null, "borrower": null, "lender": null},
  "property_address": null,
  "legal_description": null,
  "loan_amount": null,
  "purchase_price": null,
  "dates": {"executed": null, "recorded": null, "closing": null},
  "recording_info": {"book": null, "page": null, "instrument": null},
  "documentary_stamp_tax": null
}

Money fields are plain JSON numbers; dates are YYYY-MM-DD strings. The adapter was trained with no system prompt — supply only the user message. Adding a schema-describing system prompt at inference measurably degrades results (see below), so don't.

Results

Held-out test set, n = 30, base model mlx-community/Qwen3-8B-4bit, greedy decoding. "Naive" = base model, no schema given. "Schema prompt" = base model with the full schema in a system prompt. The meaningful comparison is fine-tuned (no schema prompt) vs schema-prompted baseline: the adapter reaches full schema conformance and field accuracy with a shorter prompt and no schema at inference.

Per-field accuracy:

Adding the schema prompt to the fine-tuned model is a net regression (100.00% to 95.21% overall): the adapter was trained with no system message, so injecting one at inference is off-distribution. Serve it with the bare user message.

Out-of-distribution: the 10 real forms (4096-token cap)

The only genuinely out-of-distribution data is the 10 real CFPB/FAR-BAR forms, which differ from the synthetic templates in length, layout, and content. Fine-tuned model, no schema prompt, generation cap raised to 4096 tokens (2048 was enough to truncate one dense document mid-JSON):

With only 10 documents each is worth 10 points, so these are coarse. Eight of ten were extracted with full schema conformance and all 16 fields matching the reference. The residual errors are genuine model behavior on unfamiliar documents (one value disagreement; one real structure/extraction miss), not measurement artifacts. Note the reference labels here were also produced by qwen3:32b, so this measures agreement with the label-generator on real documents, not human ground truth.

Training configuration

• Framework: mlx-lm LoRA (fine_tune_type: lora).
• Base model: mlx-community/Qwen3-8B-4bit.
• LoRA parameters: rank 8, scale (alpha) 20.0, dropout 0.0, applied to the top 16 layers.
• Optimizer: Adam, learning rate 1e-5, no LR schedule.
• Iterations: 600. Batch size 1, gradient accumulation 1, max sequence length 2048, gradient checkpointing off.
• Training seed: 0. Evaluation every 200 steps on 25 validation batches; checkpoints every 100 iterations.
• Validation loss: 1.979 (initial) to 0.216 (iter 600), monotonically decreasing and still falling at the final iteration, with no train/val divergence (no overfitting observed).
• Data split: stratified 80/10/10 by document category (train 242, valid 30, test 30). Data-generation seed 20260829.
• Memory note: batch size 2 ran out of unified memory on the training machine; resolved by reducing to batch size 1 (gradient accumulation could recover a larger effective batch without the memory cost, but was not needed).

Usage (mlx-lm)

Download the adapter and run generation, pointing --adapter-path at the folder. Supply only the user message — no system prompt.

# 1. fetch the adapter
hf download jonathonbrown/closing-doc-extraction-qwen3-8b-lora --local-dir ./closing-doc-lora

# 2. generate (base model is pulled automatically)
python -m mlx_lm.generate \
  --model mlx-community/Qwen3-8B-4bit \
  --adapter-path ./closing-doc-lora \
  --max-tokens 4096 \
  --prompt "Extract the key fields from this document as JSON:

<document text here>"

Python API:

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load(
    "mlx-community/Qwen3-8B-4bit",
    adapter_path="./closing-doc-lora",
)

doc = "<document text here>"
messages = [{"role": "user",
             "content": f"Extract the key fields from this document as JSON:\n\n{doc}"}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False)

out = generate(model, tokenizer, prompt=prompt,
               max_tokens=4096, sampler=make_sampler(temp=0.0), verbose=False)
print(out)  # a single JSON object matching the schema above

License

MIT.

Source and reproducibility

Training data pipeline, eval.py, the full RESULTS.md, and the committed per-example evaluation outputs (so every number above can be recomputed) live in the companion repository:

https://github.com/Jonathon-Brown/closing-doc-extraction