Downloads · 30 days
15
6% of all-time downloads
abhishekai/slm-125m-legal-base
slm-125m-legal-base is a text generation model from abhishekai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 125M-parameter Llama-architecture base model pretrained from scratch on a legal- and finance-heavy corpus (US case law, SEC filings, and a slice of FineWeb-Edu).
Downloads · 30 days
15
6% of all-time downloads
All-time downloads
263
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors503 MB · 100%
From the Hugging Face model README
A 125M-parameter Llama-architecture base model pretrained from scratch on a legal- and finance-heavy corpus (US case law, SEC filings, and a slice of FineWeb-Edu).
This is a base model: it does next-token prediction only. It has not been instruction-tuned, chat-tuned, or RLHF'd, and it will not follow instructions.
| Parameters | 125,848,320 (tied embeddings) |
| Architecture | Llama (LlamaForCausalLM) |
| Layers / hidden / heads | 12 / 768 / 12 (head dim 64, MHA) |
| Intermediate size | 3072 (SwiGLU) |
| Context length | 1024 |
| Vocab | 16,384 (byte-level BPE, trained on this corpus) |
| Position encoding | RoPE (theta 10,000) |
| Norm | RMSNorm (eps 1e-5) |
| Precision | bf16 autocast, fp32 master weights |
The tokenizer is custom and trained on this corpus. A 16K vocab is small by modern standards; it was chosen to keep the embedding table from dominating a 125M budget.
Scored by a Claude Sonnet judge on four axes out of 10, over 300 held-out grounded Q&A prompts, on the same scale as every other checkpoint in this project.
| mean score | 1.46 / 10 |
| 95% CI | [1.35, 1.58] |
| paired step | +2.59 to the SFT built on it (4.05), CI [+2.26, +2.91] |
This is the expected result and not a defect. A pretrained base model has no instruction tuning: prompted as a plain continuation (it has no chat template), 93% of its outputs trip a repetition detector, averaging ~1,175 characters where the SFT model answers in ~180. It continues text; it does not answer questions. The number is here so the fine-tuning step has a measured floor to stand on.
Pretraining used 2,039,072,768 tokens across 1,991,282 packed 1024-token windows, with a 1% held-out validation split (20,601,856 tokens / 20,119 windows).
| Source | Share of tokens |
|---|---|
| HFforLegal/case-law (US) | 35.1% |
| PleIAs/SEC | 42.2% |
HuggingFaceFW/fineweb-edu (sample-10BT) | 22.8% |
The mix is legal-first rather than the more common 70/20/10 web-heavy split: the two legal sources cap at roughly 2B tokens, so the pipeline takes all of both and caps web at 0.5B.
Data preparation: a 6-step deterministic cleaning chain (line filtering, non-alphanumeric ratio, minimum document length, repetition/n-gram filtering, language detection, and an OCR-quality gate applied to case law only), then MinHash near-duplicate removal, then 13-gram decontamination against LexGLUE / CaseHOLD evaluation sets (480,908 eval 13-grams).
| Tokens seen | 4.08B (2 epochs) |
| Steps | 7,778 |
| Global batch | 524,288 tokens |
| Optimizer | AdamW (beta 0.9/0.95, weight decay 0.1 on 2-D params only) |
| LR schedule | linear warmup over 200M tokens to 6e-4, cosine decay to 6e-5 |
| Grad clip | 1.0 |
| Hardware | 8x H100 |
| Throughput | ~411k tokens/s (~36% MFU) with torch.compile |
| Compute cost | ~$11.40 |
Perplexity over the entire held-out validation split (20,558,208 target tokens):
| Metric | Value |
|---|---|
| Validation loss | 2.2143 |
| Validation perplexity | 9.155 |
This is in-domain perplexity on data drawn from the same mix as training. It is not a capability benchmark, and it is not comparable to perplexity numbers computed against a different corpus or with a different tokenizer — a 16K vocab makes perplexity look lower than it would with a 32K or 50K vocab.
No downstream task benchmarks (MMLU, LegalBench, CaseHOLD, etc.) have been run.
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-base")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-base")
ids = tok("The plaintiff shall bear the burden of", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True,
temperature=0.8, top_p=0.95,
pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0], skip_special_tokens=True))
This model is a research and educational artifact — a demonstration that a coherent domain-specific base model can be pretrained end-to-end on a small budget. It is not suitable for production use, and specifically:
The model weights are released under Apache-2.0.
The training corpus draws on three sources with their own terms. None carries a share-alike or non-commercial clause, so nothing upstream constrains the license of these weights, but two require attribution:
| Source | License | Obligation |
|---|---|---|
| HFforLegal/case-law | CC-BY-4.0 | attribution |
| PleIAs/SEC | CC0-1.0 | none |
| HuggingFaceFW/fineweb-edu | ODC-By 1.0 | attribution |
This section is that attribution, and it stands whether or not those terms legally reach model weights — a question that is genuinely unsettled and largely untested in court. The common position is that weights are not a derivative work of the training data; Apache-2.0 plus visible credit is the choice that holds up under either answer.
None of this is legal advice. If you intend to use this model commercially, get your own counsel to look at it.
The full pipeline — streaming, cleaning, deduplication, decontamination, tokenizer training, packing, and distributed pretraining — runs on Modal and is reproducible. Data preparation (Phases 0-4) cost $1.73 on CPU; pretraining cost ~$11.40 on 8x H100.