slm-125m-base

SLM-125M-Base: A Legal/Financial Small Language Model

A 125M-parameter decoder-only transformer trained from scratch on a curated legal and financial corpus. Built on the LLaMA architecture with a custom 16K byte-level BPE tokenizer.

This is a base model (no instruction tuning or RLHF). It is intended as a foundation for downstream fine-tuning on legal and financial NLP tasks.

GitHub: trehansalil/slm-engineering

Model Details

Training Data

The model was trained on a legal-first data mix (~40/40/20) totaling 2.04 billion tokens after cleaning and deduplication:

Data Processing

All data underwent a rigorous cleaning and deduplication pipeline:

• 6-step deterministic cleaning: line filtering, boilerplate stripping, repetition detection, English language filtering, OCR quality gating, minimum length enforcement
• Exact deduplication via blake2b hashing on normalized text
• Near-deduplication via MinHash LSH (128 permutations, Jaccard threshold 0.7)
• Contamination stripping: 13-gram overlap removal against CaseHOLD and LexGLUE evaluation sets to prevent benchmark leakage

Training Procedure

Training Loss

Evaluation

Evaluation was conducted on a held-out 1% split (20.6M tokens) of the training corpus.

Intended Use

This model is designed as a base model for fine-tuning on domain-specific legal and financial tasks, including:

• Legal document classification
• Contract analysis and clause extraction
• SEC filing summarization
• Legal question answering
• Financial sentiment analysis

Out-of-Scope Use

• This is a small (125M) model and is not suitable as a general-purpose assistant or chatbot without significant fine-tuning
• The model has not been instruction-tuned or aligned — it may generate harmful, biased, or factually incorrect content
• Not suitable for legal advice, financial decisions, or any application requiring factual accuracy without human review

Limitations

• Model size: At 125M parameters, this model has limited capacity compared to larger LLMs. It is best suited for focused domain tasks rather than broad language understanding.
• Single epoch: The model was trained for 1 epoch over 2.04B tokens. Additional training epochs or more data may improve performance.
• Context length: Limited to 1,024 tokens. Documents longer than this must be chunked.
• English only: The model was trained exclusively on English-language text.
• Temporal cutoff: Training data reflects documents available as of mid-2026. The model has no knowledge of events after this date.
• No safety alignment: The model has no RLHF, constitutional AI, or other safety training.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Saliltrehan7/slm-125m-base")
tokenizer = AutoTokenizer.from_pretrained("Saliltrehan7/slm-125m-base")

prompt = "The court held that the defendant"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Infrastructure

The entire pipeline — data processing, tokenizer training, model pretraining, and deployment — ran on Modal's serverless GPU platform. Total compute cost: $12.36.

Citation

@misc{slm125m2026,
  title={SLM-125M-Base: A Legal/Financial Small Language Model},
  author={Salil Trehan},
  year={2026},
  url={https://huggingface.co/Saliltrehan7/slm-125m-base}
}