legalcite-support-base

legalcite-support-base

DeBERTa-v3-base fine-tuned as a 4-way classifier: given a proposition, a citation, and the cited text, predicts whether the cited text SUPPORTS, does not support (UNSUPPORTED), CONTRADICTS, or cannot be verified from (NOT_VERIFIABLE) the proposition.

Automated verification only. Not legal advice. Every citation flagged or cleared here must be independently confirmed by a licensed attorney before filing. Jurisdiction: US federal and state case law only.

This notice is printed on every output of the legalcite CLI and cannot be suppressed by any flag or config value.

Intended use

A verification aid for a legal professional checking their own citations before filing — not a legal research tool, not a source of legal conclusions, and not something that decides whether a citation exists.

This model never decides citation existence. That decision belongs entirely to eyecite + a CourtListener lookup, upstream of this model in the legalcite pipeline. A citation that doesn't resolve is labeled FABRICATED before this model is ever consulted. This model only classifies support, given text that has already been retrieved.

Do not use this model for: legal advice, drafting argument text, recommending alternative citations, or any task beyond "does this retrieved text support this proposition."

Usage

Recommended: the legalcite CLI

pip install legal-citation-check
legalcite check brief.pdf --json

This runs the full pipeline this model is one stage of — eyecite parse → CourtListener existence check → retrieval → this model classifies → mandatory notice — not just the raw classifier. It also correctly constructs the input format below for you. Prefer this over calling the model directly unless you have a specific reason not to; see the "Important" note under "Direct usage" for why.

Direct usage via transformers

This model expects a specific 3-segment input, not a plain sentence or sentence pair: [CLS] proposition [SEP] citation_string case_name [SEP] cited_text [SEP], assembled from three separately-tokenized segments (a single templated string relies on the tokenizer recognizing a literal [SEP] substring, which is fragile — this doesn't). Calling this model through a generic pipeline("text-classification", ...) on raw concatenated text will not reproduce the training-time input and will give unreliable results.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL = "vrushket/legalcite-support-base"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForSequenceClassification.from_pretrained(MODEL)
model.eval()

def build_input_ids(proposition, citation_string, case_name, cited_text, max_length=512):
    cls_id, sep_id = tokenizer.cls_token_id, tokenizer.sep_token_id
    budget = max_length - 4
    prop_ids = tokenizer(proposition, add_special_tokens=False)["input_ids"]
    cite_ids = tokenizer(f"{citation_string} {case_name}", add_special_tokens=False)["input_ids"]
    fixed_len = len(prop_ids) + len(cite_ids)
    cited_ids = tokenizer(
        cited_text, add_special_tokens=False, truncation=True, max_length=max(0, budget - fixed_len)
    )["input_ids"]
    return [cls_id] + prop_ids + [sep_id] + cite_ids + [sep_id] + cited_ids + [sep_id]

input_ids = build_input_ids(
    proposition=(
        "The right to privacy, as established in Roe v. Wade, 410 U.S. 113 (1973), "
        "is broad enough to encompass a woman's decision whether to terminate her pregnancy."
    ),
    citation_string="410 U.S. 113",
    case_name="Roe v. Wade",
    cited_text=(
        "The Court held that the right to privacy, whether it be founded in the "
        "Fourteenth Amendment's concept of personal liberty, is broad enough to "
        "encompass a woman's decision whether or not to terminate her pregnancy."
    ),
)
inputs = {
    "input_ids": torch.tensor([input_ids]),
    "attention_mask": torch.tensor([[1] * len(input_ids)]),
}

with torch.no_grad():
    logits = model(**inputs).logits

predicted_label = model.config.id2label[int(logits.argmax())]
confidence = torch.softmax(logits, dim=-1).max().item()
print(predicted_label, round(confidence, 4))
# SUPPORTED 0.9875

Important — read this before writing your own proposition text. Every training proposition was extracted as the sentence surrounding an actual citation occurrence (see ingest/citegraph.py), so it always contains the citation itself. Tested directly: a citation-free abstract restatement of the same legal point — "A state may not impose an undue burden on the right to an abortion." — classifies as UNSUPPORTED at 0.9999 confidence against this exact same evidence, where embedding the citation in the proposition (as above) correctly gives SUPPORTED at 0.99. Write propositions that include the citation itself, the way a real citing sentence would, not a clean citation-free thesis statement — see "Known limitations" below for the full diagnosis.

Or, with the package installed, reuse the exact tested implementation instead of hand-rolling the input assembly:

from legal_citation_check.models.encoder import build_input_ids, load_model_and_tokenizer

model, tokenizer = load_model_and_tokenizer("vrushket/legalcite-support-base")

What calling this model directly does not do for you

It does not decide whether a citation exists, retrieve the cited text, or check jurisdiction — that's the legalcite pipeline's job (eyecite

• CourtListener), upstream of this model. If you call this model directly, you are responsible for retrieving cited_text correctly and confirming the citation exists first; skipping that is exactly the failure mode legalcite check exists to prevent.

Training data

• 300,000 US court opinions (data/processed/cases.parquet): SCOTUS + all 13 federal circuit courts of appeals, plus the supreme and intermediate appellate courts of five large state jurisdictions (CA, NY, TX, IL, FL), 1950–2025. Stratified-sampled by court so no single court dominates the corpus.
• Sources: 63% Caselaw Access Project (CC0), 37% CourtListener's public database. Per-row provenance (source field) tracked from ingestion through to this card. Never trained on Pile of Law (CC BY-NC-SA 4.0, incompatible with this model's Apache-2.0 license).
• 355,864 labeled examples: SUPPORTED 44.5%, UNSUPPORTED 37.0%, CONTRADICTED 7.3%, NOT_VERIFIABLE 11.2%. SUPPORTED examples come only from real citing relationships (never synthesized). NOT_VERIFIABLE comes only from genuine retrieval failure (never synthesized). UNSUPPORTED/CONTRADICTED negatives are synthesized via five perturbation types (wrong case name, wrong reporter, fabricated cite, real cite/wrong proposition, real cite/wrong quote), sampled from the corpus's own real per-reporter volume/page distributions so the model can't learn a format shortcut instead of the underlying reasoning.
• Split: by citing_case, enforced in code (a citing case's examples never cross the train/test boundary) — not just a convention.
• Held out entirely, never trained on: LegalBench, LexGLUE, CaseHOLD, LegalCiteBench, LePhantomCite.

Training procedure

microsoft/deberta-v3-base, 4-way classification head, 4 epochs, DDP across 3x RTX 6000 Ada (48GB, PCIe, no NVLink), fp16, batch size 32/device, learning rate 2e-5. Input format: [CLS] proposition [SEP] citation_string case_name [SEP] cited_text [SEP], assembled from separately-tokenized segments (not a single templated string) so the special tokens are guaranteed correct regardless of tokenizer quirks. Released weights are fp16 (~380MB).

Evaluation

Held-out internal test split (53,541 examples)

The false-negative rate is the safety-critical number here, not raw accuracy: it's the rate at which an actual hallucination gets waved through as SUPPORTED, which is the failure mode that reaches a judge.

Contamination check (13-gram overlap, train vs. test)

Read this with a caveat, not as a clean number. The large gap between the two fields suggests this is driven substantially by common legal citation boilerplate — courts genuinely reuse near-identical phrasing when citing the same landmark precedents across thousands of unrelated opinions — rather than a data-pipeline leak. The class-wise F1 pattern above (CONTRADICTED meaningfully lower than the other three) is somewhat reassuring: if the held-out score were purely contamination-driven memorization, all four classes would likely sit near ceiling, not just three of them. Even so, the reported macro F1 should not be treated as a fully contamination-free number.

LegalCiteBench (external benchmark, cat4_2_case_verification)

This is a real, investigated limitation — see below.

Known limitations

1. Out-of-distribution generalization gap on propositions that don't look like training-time extracted sentences, with two distinct, directly-tested causes:

   • Citation-free propositions. Every training proposition was extracted as the sentence surrounding an actual citation occurrence (ingest/citegraph.py), so it always contains the citation. A clean, citation-free restatement of the identical legal point — "A state may not impose an undue burden on the right to an abortion." — classifies as UNSUPPORTED at 0.9999 confidence against evidence that plainly supports it; embedding the citation in the proposition text ("...as established in Roe v. Wade, 410 U.S. 113 (1973), is broad enough...") correctly flips it to SUPPORTED at 0.99. See "Usage" above for a worked example. This matters most for anyone calling this model directly, since a citation-free thesis statement is the natural way to phrase a proposition if you haven't read this card.
   • Interrogative phrasing. Predictions on LegalCiteBench's 57 resolved rows are made with high confidence (commonly 0.98–1.0) even when wrong — confident extrapolation, not the model hedging near 50/50. Isolating LegalCiteBench's trailing question-style sentence ("Can I reference X for Y?") as the proposition input caused complete mode collapse: 100% of predictions became UNSUPPORTED regardless of true label. The model has apparently learned "interrogative phrasing → default to UNSUPPORTED," a shortcut invisible on our own test set (zero question-form propositions) but decisive on LegalCiteBench's LLM-synthesized (Claude-3.5-haiku-generated) analytical-paragraph question style.

   Practically: phrase propositions as declarative sentences that include the citation itself, the way a real citing sentence would — not a question, and not a clean citation-free thesis statement.

2. Very low external-benchmark coverage. Only 0.89% of LegalCiteBench's cat4_2_case_verification rows resolved against our local 300K-case corpus index (a different, independently-drawn CAP sample), so the 49.1% figure above rests on n=57 — informative enough to diagnose the gap above, too small to be a precise external accuracy estimate on its own.

3. LePhantomCite and CaseHOLD are not integrated. LePhantomCite had no confirmed public download location when checked (an earlier AI-generated web-search summary claiming otherwise was verified to be wrong: the referenced HF org page returned 404). CaseHOLD was not attempted. Both remain open work.

4. This model never decides existence — see "Intended use" above. Existence and retrieval failures are handled entirely upstream by eyecite + CourtListener, which is itself rate-limited to 125 requests/day; citations outside the local corpus index are slow to verify at inference time.

5. NOT_VERIFIABLE is a mandatory abstain class, not a rare edge case. A verifier that guesses when it can't actually check is worse than no verifier — abstention is a feature of this model, not a shortfall.

Not legal advice

This tool assists a legal professional in checking their own work. It does not tell anyone what the law is, and does not create an attorney-client relationship. It outputs a support label and a confidence score — never a recommendation, never "you should cite X instead." Every citation flagged or cleared by this model must be independently confirmed by a licensed attorney before filing.

License

Apache-2.0. Training data: CC0 (Caselaw Access Project) and CourtListener public data, plus synthetic perturbations derived from that data. No Pile of Law content.