laya-kvp10k-noul

Laya KVP-10K — Key/Value Match Detector (noul), v2 (remediated release)

A fine-tuned Laya model (Convai Innovations, 421M, ModernBERT-large backbone) that answers one typed noul question: does a value correctly match its key label? (e.g. first_name = John → true, first_name = 1992 → false).

Fine-tuned on the IBM KVP-10K dataset via the pre-parsed community mirror alessandrorusso21/KVP10k (OCR pre-extracted; no OCR step needed).

Scope note. This is an experimental fine-tune on KVP-10K data for one very specific purpose: judging whether a key label matches a paired value inside a form-like document excerpt (English, 512-token budget). It is not a general-purpose key-value extraction or document-understanding model, and the reported numbers are claims about that narrow task only.

This is the v2 (remediated) release: document-level 80/10/10 data partitioning, mixed-class frozen test set, calib-split temperature fitting, and a frozen category-stratified release gate. It supersedes the v1 release (sothiem/laya-kvp10k-noul, pre-remediation, historical artifact).

⚠️ This is not a generative LLM

Laya is a non-autoregressive decision model: you give it a state (text/dict) plus typed questions (choice, noul, score) and it returns calibrated probabilities in a single forward pass. It never generates text. Do not use chat templates, generate(), or causal-LM losses with this checkpoint. It is loaded with laya.Agent(<dir>), not transformers.

Model details

Question definition

type: noul
id: key_value_match
instructions: "The value correctly matches the key label."
criteria:
  "false": "the value does not match the key"
  "true": "the value is consistent with the key"

State format

state = {
    "key": "first_name",
    "value": "1992",
    "document_excerpt": "<token-budgeted window around the value span>",
}

Usage

from laya import Agent

agent = Agent("<this-repo-dir-or-hf-id>")

state = {
    "key": "first_name",
    "value": "1992",
    "document_excerpt": "Name: John Smith  DOB: 1992-04-01  ...",
}
question = {
    "type": "noul",
    "id": "key_value_match",
    "instructions": "The value correctly matches the key label.",
    "criteria": {
        "false": "the value does not match the key",
        "true": "the value is consistent with the key",
    },
}
p = agent.ask(state, [question])[0]   # [P(false), P(true)]
print(p[1])                           # P(match)

Noul options render in fixed [false, true] order and must never be shuffled — p[1] is P(true).

Performance

Headline result: the frozen mixed-class test set (test_eval, n = 10,538, built only from the HF test documents, evaluated exactly once at release; the fitted temperature applied without refitting):

Per source (this checkpoint):

Notes:

• +23.8 pp accuracy over the base checkpoint (0.747 → 0.985).
• The post-training noul temperature refit (T = 0.3899, fit on the document-disjoint calib split) is what buys the calibration: identical accuracy to the raw checkpoint, NLL −78%, Brier −84%, ECE −91%.
• Mismatch recall (0.9943) slightly exceeds match recall (0.9761) — gap 0.0182, below the 0.10 warn threshold. This is the expected cost of the strict 50/50 class balance via seeded synthetic negatives; match recall is the residual weakness.
• Training curve (4 epochs, val accuracy 97.64% → 98.27%; epoch 3 selected by composite score):

Category release gate (frozen probe, 7 categories × 40)

Predefined thresholds (set before the run): accuracy ≥ 0.85, match recall ≥ 0.85, mismatch recall ≥ 0.75, min n 20. The gate FAILED on reformat (reformulated values such as case/spacing/$/comma variants of the gold value): 0.725 vs the 0.85 threshold. The other six categories pass. This is a known regression inherited from the base checkpoint (which scores reformulated values below verbatim ones); it is disclosed here rather than hidden by the strong aggregate.

Training data

• Partitioning: documents of the HF train split are partitioned before any example generator runs (seed 42): 80% train (5,486 docs, 177,282 examples) / 10% val (686 docs, 16,992) / 10% calib (686 docs, 18,090). Document sets are pairwise disjoint; every negative generator runs per partition with its donor pool restricted to that partition.
• Positives: gold (key, value) pairs from KVP-10K gts/<hash>.json (kvps_list).
• Negatives (synthetic, seeded, 50/50 class balance, never a gold pair): per positive the negative budget is split over same-format digit perturbations (50%), cross-type typed values (25%), legacy same-doc / cross-doc swaps (~25%); plus reformulation-equivalent positives (0.36 per gold), a no-excerpt prior slice (~10%, train-only, base-model-agreement filtered), and seeded Faker invoice documents (~10%, train-only).
• Document excerpts: 240-token window around the value span, built from the pre-parsed OCR words in ocrs/<hash>.json.
• Held out: the HF test/ split never appears in train/val/calib and was used only for the one-shot frozen test_eval (10,538 examples, ~50/50 classes, gold-guarded) and never for hyperparameter selection. The temperature was fit on calib only; val was used only for checkpoint selection and gates.

Training procedure

Single NVIDIA RTX 5090 (32 GB), bf16 AMP, 4 epochs:

Limitations

• reformat category fails the release gate (0.725 vs 0.85): expect reduced match recall when the value appears in a reformulated form (case/spacing/$/comma/spelled-out variants) rather than verbatim.
• Temperature clamp: the fitted noul temperature (0.3899) is stored in rl_agent_config.json, but the stock laya.Agent only accepts temperatures in [0.5, 5] and clamps it to 0.5 at load (with a RuntimeWarning). The reported test metrics are measured at T = 0.5 and are already well calibrated (ECE 0.023); applying the fitted T would require rescaling outside the Agent.
• Match recall (0.9761) is the weaker class; the model is slightly "mismatch-leaning" (bias gap 0.0182).
• English documents only (512-token budget). For non-English use convaiinnovations/laya-multilingual (1024-token budget) and re-fine-tune.
• The model judges a key/value pair in the context of a document excerpt; accuracy degrades if the excerpt does not cover the value span.
• Probabilities are calibrated on the KVP-10K-style distribution (form-like documents); out-of-domain calibration is not guaranteed.

Repository contents

Citations & credits

• Laya: Convai Innovations — https://huggingface.co/convaiinnovations/laya (Apache-2.0)
• KVP-10K: IBM Research — https://research.ibm.com/blog/kvp10k-dataset
• KVP-10K pre-parsed community mirror: https://huggingface.co/datasets/alessandrorusso21/KVP10k
• Encoder: answerdotai/ModernBERT-large
• Training/evaluation code: this repository's src/ (RLCD loop, evaluation, release gate)

License

Apache-2.0 (inherits the base model's license).