inlegalbert-sc-ner-silver

inlegalbert-sc-ner-silver

Named entity recognition for Indian Supreme Court judgments — v0.1 silver baseline.

Trained on 33,000 Supreme Court judgments (1950–2024). Recognises 13 entity types: statutes, provisions, case citations, judges, petitioners, respondents, lawyers, courts, organisations, geopolitical entities, dates, witnesses, and other persons.

Silver-only baseline. Annotations are automatically generated (regex + transformer NER + gazetteer enrichment), not human-verified. Gold-annotated v1.0 is in progress.

Usage

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="evolawyer/inlegalbert-sc-ner-silver",
    aggregation_strategy="simple",
)

text = (
    "The appellant M/S Emaar MGF Land Ltd. challenged the order under s.21 "
    "of the Consumer Protection Act, 1986, relying on (2017) 15 SCC 720."
)

for e in ner(text):
    print(e["entity_group"], "|", e["word"], "|", round(e["score"], 3))

Labels

STATUTE · PROVISION · CASE_CITATION · JUDGE · PETITIONER · RESPONDENT · LAWYER · COURT · ORG · GPE · DATE · OTHER_PERSON · WITNESS

IOB2 tag set: 27 tags total (O + B-/I- × 13).

Training

Evaluation

Evaluated on a non-overlapping held-out validation split (stride=512, ~500 documents). F1 is slightly conservative: entities that land exactly on a 512-token chunk boundary are scored as FP+FN, affecting <1% of entities.

Comparison baseline: OpenNyAI's 2022 model (RoBERTa + transition-based parser, gold-annotated) achieved 91.1% overall strict F1. This v0.1 silver baseline is not directly comparable — different test sets, different annotation quality, different corpus scope. CASE_CITATION at 97.76% already exceeds OpenNyAI's PRECEDENT score of 80.1% by +17 points. Official head-to-head comparison runs in v1.0 on the locked 25-file test set.

Data Sources

Silver labels produced by four automatic annotation pipelines merged per document:

• Metadata + header projection — case metadata JSONs mapped to character offsets via RapidFuzz; produces JUDGE, PETITIONER, RESPONDENT
• Regex — 14-pattern case citation extractor + statute/provision extractor; produces CASE_CITATION, STATUTE, PROVISION
• Transformer NER — OpenNyAI en_legal_ner_trf (InLegalBERT-based), offset-corrected; produces LAWYER, COURT, ORG, GPE, DATE, OTHER_PERSON, WITNESS
• Gazetteer — 858 Central Acts with alias resolution; adds and confirms STATUTE spans

Training data: evolawyer/indian-sc-judgments-ner-silver. The 25 locked test files are excluded.

Known Limitations

• GPE (36.56% F1) and ORG (41.34% F1) — role overlap. In Indian legal text, entities like "State of Maharashtra" or "Union of India" constantly appear as GPE, PETITIONER, RESPONDENT, or ORG depending on context. The linear token classification head cannot resolve these overlapping roles. Primary focus of v1.0 CRF head.
• Positional bias. Silver training data has repetitive header structures. Performance degrades when parties appear mid-document.
• Token fragmentation. Rare or hyphenated names can be split across subwords without a CRF sequence layer.
• Silver annotations only. Labels are automatically generated. Entity boundaries may drift at span edges.
• Pre-1990 OCR noise. Judgments from 1950–1989 vary in OCR quality.
• 25-file test set is locked. The 78.67% figure is a validation estimate, not the final benchmark score.

License

Apache 2.0. Base model law-ai/InLegalBERT is MIT licensed (compatible with Apache 2.0).

---

evolawyer.com · GitHub