Downloads · 30 days
10
17% of all-time downloads
nehabharti0802/rag-vs-finetune-legal-qlora
rag-vs-finetune-legal-qlora is a text generation model from nehabharti0802. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as mit.
A LoRA adapter over Qwen/Qwen2.5-7B-Instruct, trained on synthetic question–answer pairs derived from four Indian statutes.
Downloads · 30 days
10
17% of all-time downloads
All-time downloads
58
Public
Repo size
92.2 MB
Likes
0
Public
Click a slice to open those files.
.safetensors80.8 MB · 84%
From the Hugging Face model README
rag-vs-finetune QLoRA adapter (Indian statutes)A LoRA adapter over Qwen/Qwen2.5-7B-Instruct, trained on synthetic
question–answer pairs derived from four Indian statutes.
Hub: nehabharti0802/rag-vs-finetune-legal-qlora · Code:
github.com/NehaBharti08/rag-vs-finetune
This adapter was trained to be measured, not to be deployed. It is the experimental artifact of a 2×2 benchmark comparing retrieval against fine-tuning. It is published so the benchmark is reproducible. Read the limitations before doing anything else with it.
Used on its own, without retrieval, this adapter:
refuses only 15.0% of unanswerable questions, against the base model's 41.7% — fine-tuning made it 3× less willing to say "I don't know"
cites the correct statutory section 0.9% ± 1.0 of the time (n = 300 × 3 seeds)
cites a real but wrong section in 96.9% ± 2.0 of its answers
produces a perfectly formatted, confident-looking citation every time
It names the correct statute in 92.1% ± 1.7 of answers and the wrong section within it in about the same share. That combination is the problem. A citation to the wrong Act is obvious to any lawyer at a glance; a citation to the right Act with a plausible section number has to be looked up to be caught.
It is arguably more dangerous than the untuned base model, which names the wrong Act often enough (52.3%) to be visibly wrong. It is not measurably better at section citation either: 0.9% ± 1.0 against the base model's 0.3% is indistinguishable, and both are indistinguishable from zero.
Nothing this model produces is legal advice. Do not put it in front of anyone seeking legal information.
Synthetic QA generated from the bare text of four Indian statutes:
| Act | Year | Replaces |
|---|---|---|
| The Bharatiya Nyaya Sanhita | 2023 | Indian Penal Code, 1860 |
| The Bharatiya Nagarik Suraksha Sanhita | 2023 | Code of Criminal Procedure, 1973 |
| The Bharatiya Sakshya Adhiniyam | 2023 | Indian Evidence Act, 1872 |
| The Indian Contract Act | 1872 | — |
Source: India Code, the Government of India's official repository of central legislation.
Licensing. Reuse of the statutory text rests on §52(1)(q)(ii) of the Indian Copyright Act, 1957, which exempts reproduction of bare legislative text. This is a statutory exemption, not a licence: it is narrower than a permissive licence, carries no sublicense grant, and covers the bare text only — not commentary or headnotes. Anyone reusing this adapter inherits that constraint. Full reasoning in ATTRIBUTION.md.
The adapter weights are released under MIT, matching the code. The constraint above attaches to the corpus and to text the model reproduces from it, not to the weights themselves.
Provenance: 3,171 QA pairs (2,830 train / 341 val) over 1,090 sections.
Splits were assigned at section level before any generation, so a training
pair and an evaluation question cannot share a source passage. Verified by four
decontamination checks; see
reports/decontamination.md.
Generated by a local gemma4:e4b — deliberately not Qwen-family, since a
same-family generator would distil its own style into the student.
| Base | Qwen/Qwen2.5-7B-Instruct, 4-bit NF4, double-quant, bf16 compute |
| Method | QLoRA, rank 16, α/r = 2.0, dropout 0.05 |
| Targets | all 7 projections — q,k,v,o,gate,up,down |
| Trainable | 40,370,176 params (0.53%) |
| Schedule | lr 2e-4, cosine, 3% warmup, effective batch 8 |
| Steps | 353/epoch, packing off |
| Hardware | 1× RTX A5000 24 GB, peak 10.91 GiB reserved |
| Cost | 0.248 GPU-hours for the released checkpoint |
The released checkpoint is epoch 1, not epoch 3. Validation loss rose monotonically — 0.887 → 0.919 → 1.059 — so the model was already overfitting after one epoch. This replicates on all three seeds, with epoch-1 loss spanning just 0.8873–0.8892, so the checkpoint choice is structural rather than a lucky pick. Epoch 1 is the best checkpoint that exists, and the results above are what the best checkpoint achieves.
All headline metrics are judge-free — string matches against a registry of real acts and section numbers, not LLM-judged.
| Metric | Base | This adapter | Base+RAG | Adapter+RAG |
|---|---|---|---|---|
| Correct section | 0.3% | 0.9% ± 1.0 | 83.7% | 87.5% ± 1.4 |
| Correct act | 47.7% | 92.1% ± 1.7 | 96.0% | 98.2% ± 0.5 |
| Fabrication rate | 78.0% | 96.9% ± 2.0 | 14.7% | 12.0% ± 1.1 |
| Format valid | 99.7% | 100.0% | 100.0% | 100.0% |
| Latency p50 | 3.14s | 5.38s | 3.26s | 6.06s |
The LLM judge used for secondary metrics scored Cohen's κ = 0.452 against 100 human labels — below the pre-registered 0.60 threshold — so judged accuracy is reported as secondary only.
Forgetting (MMLU): in-domain −5.5pp, out-of-domain −2.5pp. Adaptation cost more in-domain legal capability than out-of-domain capability.
Inference cost: the adapter is used unmerged, so every forward pass pays extra LoRA matmuls. It is ~1.7× slower than the base model despite a 9× shorter prompt, and there is no query volume at which its training cost pays back. Merging would likely remove most of this; that was not measured.
professional_law and jurisprudence, so it measures whether general legal
reasoning survived, not whether Indian statutes were learned.@software{rag_vs_finetune_2026,
title = {RAG vs QLoRA fine-tuning: a 2x2 factorial benchmark on Indian statutes},
author = {Bharti, Neha},
year = {2026},
url = {https://github.com/NehaBharti08/rag-vs-finetune}
}
Independent student project. Not affiliated with or endorsed by the Government of India.