reconforge-recon-lora

ReconForge Recon: financial reconciliation verdict model

LoRA adapter on mlx-community/Qwen3-1.7B-4bit. It takes one ledger entry and one bank statement entry and returns a JSON verdict: MATCH, EXCEPTION (type, severity), or ESCALATE. Trained in about 100 minutes on an Apple M5. Parse rate is 1.000 on 800 ReconEval v0.1.0 tasks.

The eval is why this adapter is published. Severity-weighted recall (R_w) scores the exception subset only. Flag precision and F1 score any non-MATCH prediction over all 800 tasks, including the 419 clean pairs. Report both. Gold tasks: caiotheodoro/recon-eval.

This checkpoint: accuracy 0.805, R_w 0.901, HIGH-severity recall 1.000, flag F1 0.824. Four training seeds, same data and hyperparameters, scored R_w 0.7457 / 0.8198 / 0.8353 / 0.9451. This file is the 0.9451. Across-seed SD is 0.0823; a retrain should land nearer 0.80. The other three runs miss HIGH-severity exceptions (operational cost 0.150 / 0.267 / 0.167; this run 0.000).

The training prompt is part of the weights. A prompt that defines exception classes instead of only naming them moves accuracy from 0.805 to 0.434 and flag precision from 0.824 to 0.508, while R_w rises to 0.940 because the model starts flagging clean pairs. Use the training prompt. Pair R_w with flag precision.

Threats to validity and per-seed results are in the source repo.

This adapter and the gold tasks live in the ReconForge collection. Examples (not live data): examples/match.json, examples/amount_mismatch.json.

Quick start

# Block 1: load the adapter
# Requires: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11+ (pinned to this repo's
# training/eval environment; see reconforge/model/uv.lock)
from huggingface_hub import snapshot_download
from mlx_lm.lora import load

adapter_dir = snapshot_download("caiotheodoro/reconforge-recon-lora")
model, tokenizer = load(
    "mlx-community/Qwen3-1.7B-4bit",
    adapter_path=adapter_dir,
    tokenizer_config={"trust_remote_code": True},
)

# Block 2: build the prompt
import json

SYSTEM_PROMPT = """You are ReconForge, a financial back-office reconciliation operations engine. \
You reconcile a single ledger entry against a single bank statement entry and return a structured verdict.

Your output MUST be exactly one JSON object with these keys:
- "verdict": "MATCH" | "EXCEPTION" | "ESCALATE"
- "exception_type": null or one of AMOUNT_MISMATCH, FX_CONVERSION_ERROR, \
BENEFICIARY_MISMATCH, COUNTERPARTY_MISMATCH, VALUE_DATE_MISMATCH, MISSING_MESSAGE, \
DUPLICATE, FIELD_CORRUPTION, PARTIAL_MATCH
- "severity": "LOW" | "MEDIUM" | "HIGH"
- "confidence": float in [0, 1]
- "reason": short reason, under 10 words
- "resolution": one of "auto-adjust", "escalate", "reject", "rebook", "flag-review"
"""

user_prompt = """Reconcile the following ledger entry against the bank statement.

LEDGER ENTRY:
{
  "amount": "10000.00", "ccy": "USD", "counterparty": "Acme Corp",
  "beneficiary": "Acme Corp", "value_date": "2024-01-15",
  "message_type": "MT300", "ref": "OUR-REF-000001"
}

BANK STATEMENT:
{
  "amount": "9999.50", "ccy": "USD", "counterparty": "Acme Corp",
  "beneficiary": "Acme Corp", "value_date": "2024-01-15",
  "message_type": "MT940", "ref": "CP-PAY-000001"
}

Return the verdict JSON object only."""

# Block 3: generate
from mlx_lm.generate import generate

for attr in ("has_thinking", "enable_thinking"):
    if hasattr(tokenizer, attr):
        setattr(tokenizer, attr, False)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_prompt},
]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False, return_dict=False
)
response = generate(
    model, tokenizer, prompt, max_tokens=128,   # single greedy sample --
                                                 # a minimal usage example,
                                                 # NOT the eval config
)
verdict = json.loads(response)
print(f"{verdict['verdict']} / {verdict['exception_type']} / {verdict['severity']}")

This Quick start uses one greedy sample. It is a minimal usage example, not the eval configuration. The Results below were produced with self-consistency sampling (5 samples, temp=0.6, majority vote); see the Run configuration table. A single greedy sample will not reproduce the reported numbers.

The model is trained in non-thinking mode (enable_thinking=False) and emits ~38 tokens per verdict on average. Leaving thinking enabled changes the output format and breaks the parse.

Results

Eval set: ReconEval v0.1.0, 800 held-out tasks, seed 777, exact-overlap contamination 0/800 (near-duplicate rate 3.6% at Jaccard ≥ 0.8; see recon-eval).

Selection caveat. Every checkpoint and configuration choice behind these results (run-1 vs the B2 rebalanced-mix run, ×3 vs ×5 self-consistency sampling) was made by comparing on this same seed-777 800-task set that the headline metrics are then reported on. It was never held out from selection. A newly designated seed (999) is frozen as the test set: its task signatures are committed in docs/validation/frozen-test-seed-999-signatures.json and nothing has been scored against it. Selection discipline going forward is in docs/TRAINING.md (Selection policy).

All CIs are 95% bootstrap intervals, 10,000 resamples, seed 11, over the same 800-task set per model (reconforge/model/scripts/intervals.py for accuracy / R_w / HIGH recall; reconforge/model/scripts/rescore_flag_metrics.py for flag precision / F1 / severity-weighted cost).

R_w is severity-weighted recall over the exception subset only (381 of 800 tasks). A "flag" is any predicted verdict ≠ MATCH, scored over all 800 tasks. R_w cannot see false positives on clean pairs: a degenerate always-ESCALATE model scores R_w 0.6673 on this benchmark at accuracy 0.0 (docs/BENCHMARK.md). Parse failures count as misses and stay in the denominator (reconforge_model/metrics.py).

DeepSeek has zero false positives on the 419 clean pairs (flag F1 0.904 [0.880, 0.926]). This run over-flags 67 clean pairs (flag F1 0.824 [0.793, 0.853]). Severity-weighted cost (cost_esc = 1.0 per review, cost_missed_high = 5.0 per missed HIGH; metrics.severity_weighted_cost defaults) is 0.000 on this run (no escalations, no missed HIGH) and 0.013 for DeepSeek.

Paired bootstrap (same 800 tasks, 10,000 resamples):

Run configuration:

Full machine-readable run configs: docs/validation/runconfig.json in the source repo.

ECE for ReconForge Recon is 0.0875 (self-consistency ECE, from the champion eval run). DeepSeek and base Qwen do not use self-consistency sampling in this eval, so a comparable ECE was not computed for them here. Reported as not available rather than estimated.

Matched prompt (exploratory)

The table above uses the training prompt (v1) and unmatched sample budgets (this adapter ×5, DeepSeek ×1). Giving both models a prompt that defines exception classes, at ×5, is a different experiment: it measures prompt lock-in, not a published head-to-head.

On seed 777, PARTIAL_MATCH surface pairs overlap training completely (fraction 1.0), which favours this adapter. The comparison is exploratory; the pre-registered confirmatory pool is seed 999 (docs/BENCHMARK.md, docs/validation/matched-comparison.json).

The rubric prompt is the same change described in the lede: this adapter's accuracy and flag precision drop; R_w rises because it over-flags clean pairs. That is why R_w is paired with flag F1.

Per-exception recall (ReconForge Recon)

Exact verdict+type match recall, with 95% bootstrap CI, grouped by the expected exception type:

R_w = 0.901 with two recall figures below 0.3 (FIELD_CORRUPTION, DUPLICATE) because those types carry the lowest weights. Without the weight column the headline looks inflated. At n = 31–52 the CIs are wide: COUNTERPARTY_MISMATCH 0.865 sits in [0.744, 0.970]. Treat single decimal-place differences between runs at this sample size as noise.

Training

Compute: Apple M5, 16 GB unified memory. Wall time ~100 min (740 iterations). Peak memory 3.346 GB. No GPU cluster, no distributed training. macOS version was not recorded.

Framework versions: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11.15 (from reconforge/model/uv.lock).

CO2: CO2 was not measured. About 100 minutes of M5 package power on a laptop-class chip; co2_eq_emissions was not computed.

Training data: 3,198 synthetic reconciliation pairs (train) + 802 (val), from generation seed 101 (regenerating the pool at seed 101 reproduces data/train.jsonl and data/val.jsonl byte-for-byte; seeds 7 and 777 do not). Stratified split by (difficulty decile, exception type) under split seed 7. Three seeds are in play and were previously conflated: generation seed 101, split seed 7, training-run seed 7; see ../docs/TRAINING.md. Contamination guard: SHA-256 field-level pair signatures, zero exact overlap between train and eval (re-confirmed under generation seed 101: overlap 0.0); see recon-eval for the full audit including the near-duplicate rate this exact-hash check does not cover.

Ablation: DUPLICATE recall across two checkpoints

(Flag precision / F1 for the b2 run was not re-scored; outside issue #8's scope. The champion ×5 run's flag F1 is 0.824 [0.793, 0.853]; see Results.)

Both runs used almost identical training-pair counts (3,198 vs 3,201), so this pair does not isolate a data-scale effect. b2 also differs in checkpoint (590 vs 700 persisted iterations) and eval self-consistency (x3 vs x5), both of which move R_w and accuracy on their own. At comparable data volume, DUPLICATE recall stayed at or near zero across both checkpoints (0/31 and 1/31), while every other metric moved between the two runs. That is consistent with the discriminating signal for DUPLICATE (statement.reference == ledger.reference while all other fields match) being a single exact-equality predicate the model does not reliably attend to, rather than a data-volume problem. It does not prove that. A clean data-scaling ablation (same checkpoint, different train-set sizes) has not been run; treat "architecture/scale limit, not data limit" as a hypothesis this ablation is consistent with. DUPLICATE detection is free and exact with a deterministic reference-equality pre-check; see Out-of-scope use.

Uses

Direct use: classifying ledger↔statement reconciliation pairs into MATCH / EXCEPTION(type, severity) / ESCALATE with a structured JSON verdict.

Downstream use: as one scorer inside a larger reconciliation pipeline, behind a rule pre-check and in front of a human review queue.

Out-of-scope use (read this before deploying):

• Duplicate detection: recall is 0.000 on the eval set. Use a deterministic reference-equality check. Routing duplicates to this model means missing (nearly) all of them.
• Uncertainty signalling: the model emits ESCALATE zero times in this eval. It has no demonstrated way to say "unsure." Every HIGH-severity verdict must route to human review unconditionally. That is a deployment requirement.
• Live financial data: trained on synthetic pairs only. Performance on real ledgers is unmeasured.
• Autonomous judging: the production recalibration judge is DeepSeek + JUDGE_SYSTEM_PROMPT (the rubric-extended prompt), which reaches Cohen's kappa 0.9037 against the oracle on the 100-task golden set. Its 95% bootstrap CI over that set is [0.828, 0.968], whose lower bound is below the 0.85 threshold, so the point estimate clears the bar but the data at n=100 cannot distinguish this judge from one sitting exactly at it. This adapter is not that judge. Asked to self-judge under the same rubric prompt it was never trained on, its kappa is 0.367 (a regression from 0.736 on the bare prompt; extended rubric text is off-distribution for a 1.7B fine-tuned on one fixed prompt; see docs/DECISIONS.md C2). Do not use this adapter for unsupervised judging.
• Any non-reconciliation domain: single-domain by construction; does not transfer to payment repair or settlement.

Limitations and bias

1. Synthetic data only. Generated from a parameterized oracle, not sampled from production systems. Real reconciliation traffic has correlations, seasonality, and counterparty long tails this data does not model. Treat the absolute numbers as an upper bound.
2. DUPLICATE recall ~0. See Ablation above; not fixable by the data-volume change tested so far.
3. Zero escalations observed in this eval run; see Out-of-scope use.
4. Judge calibration gap. The production judge (DeepSeek + JUDGE_SYSTEM_PROMPT) is at kappa 0.9037 [0.828, 0.968] vs oracle, above the 0.85 bar on the point estimate, with an interval that includes values below it, so "calibrated" is not yet demonstrable at n=100. This adapter self-judging under the same rubric is at kappa 0.3672 [0.270, 0.467] (see Autonomous judging above and docs/DECISIONS.md C2). ECE 0.0875 self-consistency; treat any single-sample deployment as uncalibrated relative to this number, which was measured under 5-sample self-consistency.
5. Single-domain. No transfer evidence to payment repair or settlement.
6. Generator bias. The eval set inherits every bias of the generator, including its exception-type mix and difficulty distribution. A model tuned to that generator will look better than it is on any other distribution. The near-duplicate audit (3.6% at Jaccard ≥ 0.8) bounds train/eval leakage; it says nothing about generalization beyond the generator's distribution.

Card metadata

• Authors: Caio Theodoro
• Contact: via the HF repo discussions tab
• Repository: https://github.com/caiotheodoro/reconforge

Synthetic data. All tasks are generated, not drawn from live financial systems. No real counterparties, account identifiers, or personal data are present.

Not production-validated. Nothing here is financial advice or a validated control. Any deployment touching real money requires independent validation and a human-in-the-loop review path for high-severity cases.

Citation

@misc{theodoro2026reconforge,
  title  = {ReconForge: Severity-Weighted Evaluation for Financial Reconciliation Agents},
  author = {Caio Theodoro},
  year   = {2026},
  url    = {https://github.com/caiotheodoro/reconforge},
  note   = {LoRA adapter for Qwen3-1.7B, Apache-2.0}
}