Downloads · 30 days
10
18% of all-time downloads
Kotichitturu/gemma-2-2b-legal-qa-v2
gemma-2-2b-legal-qa-v2 is a machine learning model from Kotichitturu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as gemma.
Fine-tuned for closed-book Q&A on US case law, SEC filings, and FineWeb-Edu (≈40/40/20 source mix, matching the pretraining distribution).
Downloads · 30 days
10
18% of all-time downloads
All-time downloads
55
Public
Parameters
2.6B
5.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.2 GB · 99%
From the Hugging Face model README
Fine-tuned for closed-book Q&A on US case law, SEC filings, and FineWeb-Edu (≈40/40/20 source mix, matching the pretraining distribution).
eval_v2Scored on a held-out benchmark built from fresh passages (different sampling seed), decontaminated against every training set, and frozen with a sha256 receipt. Judge = claude-sonnet-5 (a different model family from both this model and the data pipeline's teacher), rubric 0–100.
| Metric | Value |
|---|---|
| Judge score | 35.5/100 (95% CI 33.2–37.7) |
| Correct (vs reference) | 31.2% |
| Grounded | 24.7% |
| Refusal recall (answer absent → refused) | 0.0% (n=296) |
| Over-refusal (answer present → refused) | 0.0% (n=366) |
| Numeric match | 96.0% (n=148) |
| Token F1 vs reference | 23.1% (n=1030) |
| Items scored | 1325 |
| Task | Judge /100 | Token F1 | n |
|---|---|---|---|
| Closed-book QA (no context — recall from weights) | 16.1 | 7.8% | 380 |
| Instruction following (no context) | 19.5 | 32.9% | 283 |
| RAFT (answer IS in the retrieved context — extraction) | 94.5 | 31.4% | 366 |
| Refusal (answer is ABSENT — must decline) | 2.5 | — | 296 |
Read the refusal row against the RAFT row. A high RAFT score with a low refusal score means the model answers well when the answer is present and fabricates when it is not — the two are the same behaviour measured twice.
Gold answers are model-authored (
gemini-3.1-pro) from a source passage. These measure agreement with the reference, not truth.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Kotichitturu/gemma-2-2b-legal-qa-v2")
model = AutoModelForCausalLM.from_pretrained(
"Kotichitturu/gemma-2-2b-legal-qa-v2", torch_dtype="bfloat16",
attn_implementation="eager", # Gemma 2 soft-capping is silently WRONG under sdpa
).eval()
SYSTEM = 'You are a legal and financial expert. Answer accurately and concisely. If unsure, say so.'
msgs = [{"role": "user", "content": SYSTEM + "\n\n" + "<your question>"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
Teacher-generated and judge-filtered, not scraped:
gpt-4o-mini, then a regex prefilter (self-containment, anchoring)
removes ~20% for free before any paid judging.gemini-3.1-flash-lite for faithfulness and correctness — a
different model family from the teacher, so it cannot reward its own idiom.