fine_tuned_Qwen2.5-7B-Instruct_CUADv1

Qwen2.5 7B — Contract Clause QA

Supervised instruction-tuning of Qwen2.5 7B for contract clause identification and extraction across 41 legally significant clause types from the CUAD benchmark.

---

Overview

Models are fine-tuned to produce a structured response: a binary Yes/No decision on clause presence, followed by the verbatim quotation of the relevant clause. This reframes CUAD's span-extraction task as an instruction-following problem, allowing the model to leverage parametric legal knowledge while remaining grounded in the provided contract excerpt.

• Base model: Qwen2.5 7B
• Dataset: CUAD — 510 annotated commercial contracts, 41 clause types
• Adaptation: LoRA and QLoRA (r=16, α=16, all linear layers)

---

Performance

Composite metric:

Overall = (0.4 × Token_F1 + 0.6 × BERTScore) × Accuracy

BERTScore uses Legal-BERT as the scoring backbone with IDF weighting and baseline rescaling.

Key finding: QLoRA achieves near-parity with full-precision LoRA at significantly lower GPU memory cost — confirming that 4-bit base weight compression does not meaningfully degrade clause identification quality at the 7B scale.

---

Fine-tuning gain vs zero-shot

At the 7B scale, SFT refines behaviour qualitatively — teaching the model to ground responses in the specific contract text rather than generalised legal knowledge, producing more faithful and legally precise extractions.

---

Training setup

---

Known Limitations

Temporal / date clauses

Subword tokenisation fragments date strings, causing models to identify the correct sentence but predict a generic label (e.g. "Effective Date") rather than the precise date token. This is a tokeniser-level limitation that fine-tuning alone cannot resolve.

Autoregressive repetition loop

Party name extraction can trap the model in a token-repetition loop until the context limit is reached, particularly when the reference contains multiple semicolon-separated entity names.

Exposure bias / cascade errors

A wrong Yes/No gate token propagates forward with no corrective signal — the model can confabulate a plausible but incorrect clause extraction (e.g. a hallucinated date) with full confidence.

---

Evaluation notes

• Length bias is present in all fine-tuned models (negative Pearson correlation between reference length and scores). The 7B QLoRA large-batch variant shows the weakest bias (r = −0.133 / −0.119).
• BERTScore floor: a rescaled score near zero indicates semantic orthogonality, not a random prediction. A corpus-specific CUAD baseline would provide a more appropriate rescaling anchor.
• Batch size effect: larger batch sizes produce a weaker 7B QLoRA model, suggesting gains are driven by the number of gradient updates rather than update quality.