qwen-contract-clause-extractor

qwen-contract-clause-extractor

QLoRA fine-tune of Qwen2.5-1.5B-Instruct that extracts 12 key clause types from commercial contracts into structured JSON.

Model Details

• Base model: Qwen/Qwen2.5-1.5B-Instruct
• Fine-tuning method: QLoRA (4-bit NF4 quantization + LoRA adapters)
• Task: Structured clause extraction (contract text → JSON)
• Language: English
• License: Apache 2.0 (inherited from base model)

Model Description

Given raw contract text, the model returns a single JSON object with 12 fields, using null for any clause category not present in the contract:

parties, agreement_date, effective_date, governing_law, termination_for_convenience, cap_on_liability, non_compete, exclusivity, expiration_date, license_grant, document_name, renewal_term

These map to 12 of the 41 clause categories in CUAD (Contract Understanding Atticus Dataset), chosen for being common across most commercial contract types.

Intended Uses

• Drafting-assistance / first-pass review tooling that flags key clauses in a contract for a lawyer to verify
• Structured-extraction portfolio/benchmark example for small-model QLoRA fine-tuning

Not intended for: standalone legal advice, or any use where extracted output is relied upon without human review. This model has a non-trivial hallucination rate on absent fields (see Limitations).

How to Use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "meharshh/qwen-contract-clause-extractor"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id, quantization_config=bnb_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)

instruction = (
    "Extract the following fields from this contract. "
    "Return valid JSON only, with null for any field not present.\n"
    "Fields: parties, agreement_date, effective_date, governing_law, "
    "termination_for_convenience, cap_on_liability, non_compete, exclusivity, "
    "expiration_date, license_grant, document_name, renewal_term\n\n"
    "Contract text:\n<your contract text here>"
)
messages = [{"role": "user", "content": instruction}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training Data

CUAD — 510 real commercial contracts with expert legal annotations, originally SQuAD-style span extraction. Reshaped into one instruction→JSON example per contract (285 train / 61 validation / 62 test).

Training Procedure

Evaluation

Evaluated on a 62-example held-out test set. Metric: per-field exact match (case-insensitive, whitespace-normalized) computed only over outputs that parsed as valid JSON, plus overall JSON validity rate.

Fine-tuning improved 8 of the 12 fields, with a net +9.8 percentage point gain in overall field accuracy.

Loss curve

Train loss decreases steadily across all 3 epochs, but eval loss is nearly flat after epoch 1 — the model is data-limited rather than undertrained; more epochs on this dataset size wouldn't meaningfully help, more/varied training examples would.

Example output

Input: a 1996 technology license agreement.

{
  "parties": "ENTRUST TECHNOLOGIES INC.",
  "agreement_date": "31 December, 1996",
  "effective_date": "31 December, 1996",
  "governing_law": "This Agreement will be governed by and construed in accordance with the laws of the Province of Ontario without regard to conflicts of law principles.",
  "termination_for_convenience": null,
  "cap_on_liability": "IN NO EVENT SHALL EITHER PARTY BE LIABLE TO THE OTHER FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES ARISING OUT OF THIS AGREEMENT, EVEN IF SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.",
  "non_compete": null,
  "exclusivity": null,
  ...
}

Qualitative error analysis

Manual review of the 4 regressed fields (governing_law, cap_on_liability, termination_for_convenience, expiration_date) shows the exact-match metric understates real quality:

• Paraphrase mismatches: e.g. ground truth "...without reference to conflict of law principles" vs. model output "...without giving effect to principles of conflicts of law" — same legal meaning, scored as wrong by exact string match.
• Valid-but-different clause selection: contracts often contain multiple sentences relevant to a category (e.g. cap_on_liability); the model sometimes extracts a different, equally valid sentence than the one CUAD's annotators chose as ground truth.
• Genuine failure mode: when a field is truly absent (null in ground truth), the model occasionally still generates a confident-sounding but fabricated clause instead of returning null.

Limitations

• Exact-match evaluation likely understates true quality: manual review found several "incorrect" predictions were paraphrases or alternate valid clause selections rather than genuine errors.
• Some fields (non_compete, exclusivity, renewal_term) had under 40% non-null coverage in training data, limiting their ceiling.
• The model occasionally generates a plausible-sounding but fabricated clause for fields that are genuinely absent, instead of returning null — outputs should always be reviewed by a human before use.
• Covers only 12 of CUAD's 41 labeled clause categories.