nightwing-14b-cuad-extractive

nightwing-14b-cuad-extractive

Paper (DOI) | Code + harness (GitHub) | The experiment write-up | The v1 story | Author on LinkedIn

LoRA adapter (+ trained extractive QA head) on Qwen/Qwen3-14B for contract clause extraction on CUAD (Contract Understanding Atticus Dataset, 41 clause categories). Given a contract window and a category question, the model predicts the start/end of the supporting span, or "absent".

Official CUAD test AUPR: 0.3885 (full 102-contract test split, official evaluator, test touched exactly once by this dev-selected checkpoint).

Honest scoreboard (same harness, same metric, same test split)

• Beats gpt-4o in 25/40 categories and gpt-5.2 in 22/40; beats all four frontier models simultaneously in 6 (e.g. Agreement Date 0.829 vs best frontier 0.168).
• The v1 -> v2 delta (+9.8 pts) comes from task framing alone: generation -> extraction, same base model, same data, ~$32 of A100 time, one epoch.
• A pre-registered prediction that this run clears 0.50 missed; the deficit is the deliberately lean recipe (1 epoch, 2:1 negative subsampling, no tuning). Details, scaling curve (0.5B-14B), and every raw prediction file: github.com/ashish24142/nightwing.

IMPORTANT: loading correctly

Qwen3ForQuestionAnswering names its backbone transformer.* while the base checkpoint stores model.*, so a vanilla AutoModelForQuestionAnswering.from_pretrained("Qwen/Qwen3-14B") silently initialises a random backbone and this adapter will produce garbage. Load the backbone explicitly (or use harness/extractive.py::load_qa_model from the repo, which asserts it):

import torch
from transformers import AutoModel, AutoModelForQuestionAnswering, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen3-14B"
qa = AutoModelForQuestionAnswering.from_pretrained(BASE, dtype=torch.bfloat16)
src = AutoModel.from_pretrained(BASE, dtype=torch.bfloat16)
missing, _ = qa.transformer.load_state_dict(src.state_dict(), strict=False)
assert not [m for m in missing if "embed" in m or "layers" in m]
del src

model = PeftModel.from_pretrained(qa, "ashish24142/nightwing-14b-cuad-extractive")
tok = AutoTokenizer.from_pretrained("ashish24142/nightwing-14b-cuad-extractive")
model.eval().to("cuda")

Inference over long contracts uses 512-token sliding windows (stride 128) with truncation="only_second"; span confidence for PR-curve scoring is sigmoid(best_span_score - min_null_score). The exact windowing, postprocess and official-metric scoring code is in the repo (harness/extractive.py, harness/eval_extractive.py), using it is the recommended path and reproduces 0.3885 exactly.

Training

• LoRA r16, alpha 32, dropout 0.05, all attention+MLP projections; randomly initialised qa_outputs head trained via modules_to_save.
• bf16, effective batch 16, lr 2e-4 cosine, 2000 steps (~1 epoch), gradient checkpointing; single A100 80GB, ~2.6 h.
• Data: CUAD train split only, 368 contracts (40-contract dev carve-out excluded), zero train/test contamination asserted in code at every launch.
• Trained only on CUAD ground truth, never on frontier model outputs.

Intended use & limitations

Research benchmark artifact for legal-NLP extraction. Not a lawyer, not legal advice, not production-ready: overall accuracy trails frontier models by 17 pts, recall at high thresholds is weak (p@80recall = 0), and the head predicts one span per window while some CUAD questions have multiple gold spans.

Citation

If you use this model or the benchmark results, cite the paper:

@misc{singh2026nightwing,
  author       = {Singh, Ashish Kumar},
  title        = {Task Framing Outweighs Model Scale in Contract Clause
                  Extraction: Pre-Registered Experiments with Frontier LLMs
                  and Small Fine-Tuned Specialists on CUAD},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21530247},
  url          = {https://doi.org/10.5281/zenodo.21530247}
}

CUAD dataset: Hendrycks et al., 2021 (arXiv:2103.06268), CC BY 4.0, The Atticus Project. Base model: Qwen3-14B (Apache-2.0).