Downloads · 30 days
4
6% of all-time downloads
abdullah693/pakistan-legal-qa-llama70b
pakistan-legal-qa-llama70b is a text generation model from abdullah693. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as llama3.3.
A LoRA fine-tune of meta-llama/Llama-3.3-70B-Instruct specialising in Pakistani law, trained on a curated dataset of 5,116 question–answer pairs grounded in primary statutory text.
Downloads · 30 days
4
6% of all-time downloads
All-time downloads
63
Public
Repo size
3.3 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3.3 GB · 99%
From the Hugging Face model README
A LoRA fine-tune of meta-llama/Llama-3.3-70B-Instruct specialising in Pakistani law, trained on a curated dataset of 5,116 question–answer pairs grounded in primary statutory text.
Given a legal question, the model produces:
PPC s.302, CONST art.184(3), QSO art.17)| Domain | Statute | QA pairs |
|---|---|---|
| Criminal | Pakistan Penal Code (PPC) | 2,215 |
| Constitutional | Constitution of Pakistan 1973 | 1,957 |
| Evidence | Qanun-e-Shahadat Order 1984 (QSO) | 742 |
| Civil Procedure | Code of Civil Procedure (CPC) | 143 |
| Family | Muslim Family Laws Ordinance 1961 (MFLO) | 59 |
| Total | 5,116 |
All answers are grounded in verbatim provision text extracted from official Pakistani statutes. A "hard slice" of 110 test questions covers Pakistani-distinctive doctrine (qisas/diyat, Qanun-e-Shahadat evidence rules, Muslim family law) where general-purpose models most commonly confuse Pakistani law with Indian or British law.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
base = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.3-70B-Instruct",
quantization_config=bnb_config,
device_map="auto"
)
model = PeftModel.from_pretrained(base, "abdullah693/pakistan-legal-qa-llama70b")
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
question = "Under PPC s.302, what is the punishment for qatl-i-amd?"
inputs = tok.apply_chat_template(
[{"role": "user", "content": question}],
return_tensors="pt", return_dict=True, add_generation_prompt=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Reasoning: <short chain-of-thought grounded in the statutory text>
Answer: <direct answer>
Citation: <canonical citation e.g. PPC s.302>
| Parameter | Value |
|---|---|
| Base model | meta-llama/Llama-3.3-70B-Instruct |
| Method | LoRA (PEFT) |
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0.0 |
| Target modules | all-linear — q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Epochs | 2 (134 steps) |
| Final train loss | 0.675 |
| Final eval loss | 0.625 (monotonically decreasing across all checkpoints) |
| Trained via | Adaption AutoScientist / Together AI |
Because the adapter targets all linear layers (including the MLP/FFN blocks where factual associations are stored), it can encode Pakistani legal knowledge rather than just output formatting. Eval loss was still declining at the end of epoch 2.
The training data was built from licence-clean Pakistani legal corpora:
Each QA pair includes a verbatim provision excerpt as grounding context plus a chain-of-thought reasoning trace generated and verified by Claude.
Dataset: abdullah693/adaption-pakistan-law-qa-pairs
LoRA weights: CC-BY-4.0. Subject to Meta's Llama 3.3 Community Licence.