Downloads · 30 days
5
1% of all-time downloads
Sarath569/slm-125m-legal-base
slm-125m-legal-base is a text generation model from Sarath569. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 125.8M-parameter Llama-style decoder-only language model pretrained from scratch on a cleaned, deduplicated, decontaminated corpus of US case law, SEC filings, and educational web text (~2.04B unique tokens; legal-f…
Downloads · 30 days
5
1% of all-time downloads
All-time downloads
740
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors503 MB · 100%
From the Hugging Face model README
A 125.8M-parameter Llama-style decoder-only language model pretrained from scratch on a cleaned, deduplicated, decontaminated corpus of US case law, SEC filings, and educational web text (~2.04B unique tokens; legal-first mix).
| Architecture | Llama (RoPE, SwiGLU, RMSNorm, tied embeddings) |
| Parameters | 125,848,320 |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context length | 1024 |
| Vocab (byte-level BPE) | 16,384 |
| Precision | bf16 |
| Tokens seen | 4.08B (2 epochs over 2.04B) |
| Optimizer steps | 7,778 |
| Optimizer | AdamW (β=0.9/0.95, wd=0.1, grad-clip=1.0) |
| LR schedule | cosine 6e-4 → 6e-5, 200M-token warmup |
| Hardware | 8× H100 (data-parallel) |
| Final val loss / ppl | 2.3307 / 10.29 |
HFforLegal/case-law) — ~722M tokens (35%)PleIAs/SEC) — ~869M tokens (42%)HuggingFaceFW/fineweb-edu) — ~469M tokens (23%)Decontaminated against CaseHOLD / LexGLUE (case_hold) with 13-gram overlap filtering.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("Sarath569/slm-125m-legal-base")
model = AutoModelForCausalLM.from_pretrained("Sarath569/slm-125m-legal-base", torch_dtype=torch.bfloat16)
ids = tok("The plaintiff shall bear the burden of", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, top_k=50, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
This is a small (~126M) base model, not instruction-tuned or aligned. It will produce
fluent but unreliable text, can hallucinate legal/financial facts, and must not be used
for legal or financial advice. Reserved chat tokens (<|user|>, <|assistant|>, <|system|>)
are present in the tokenizer for downstream fine-tuning.