Downloads · 30 days
19
2% of all-time downloads
rahulreddyhanu/slm-125m-legal-financial
slm-125m-legal-financial is a text generation model from rahulreddyhanu. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal- and financial-domain corpus. Not fine-tuned from any existing base model — tokenizer and weights are both trained from zero.
Downloads · 30 days
19
2% of all-time downloads
All-time downloads
918
Public
Parameters
126M
252 MB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors252 MB · 100%
From the Hugging Face model README
A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal- and financial-domain corpus. Not fine-tuned from any existing base model — tokenizer and weights are both trained from zero.
Streamed, cleaned, deduplicated, and decontaminated from three public HuggingFace datasets:
| Source | HF dataset | Tokens | Share |
|---|---|---|---|
| Case law | HFforLegal/case-law (US) | ~715M | ~35% |
| SEC filings | PleIAs/SEC | ~860M | ~42% |
| Web (educational) | HuggingFaceFW/fineweb-edu | ~464M | ~23% |
| Total | ~2.04B |
Legal-first mix (not 70/20/10 — the two legal sources only hold ~2B clean tokens total, so all of both were taken and a small web slice added for fluency).
Cleaning: line-level noise/boilerplate filtering, repetition filter, English-language
gate, OCR-garble gate (case-law only, dictionary-based).
Dedup: MinHash near-dup removal (case-law) + exact-hash dedup (all sources).
Decontamination: documents overlapping 13-grams from casehold/LexGLUE
case_hold eval sets were stripped, so downstream legal-QA benchmarks (e.g.
CaseHOLD) are not contaminated.
torchrun, bf16, torch.compile, flash-SDPA, fused AdamWHeld-out validation perplexity (1% split, never trained on):
| Domain | Val perplexity |
|---|---|
| SEC (financial) | 6.32 |
| Case law (legal) | 11.21 |
| Web (fineweb-edu) | 29.00 |
| Overall | 12.71 |
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("rahulreddyhanu/slm-125m-legal-financial")
model = AutoModelForCausalLM.from_pretrained("rahulreddyhanu/slm-125m-legal-financial")
ids = tok("The plaintiff shall bear the burden of proving", return_tensors="pt")
ids.pop("token_type_ids", None)
out = model.generate(**ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))