Downloads · 30 days
4
1% of all-time downloads
lekhamenon28/legal-slm-125m-2ep
legal-slm-125m-2ep is a text generation model from lekhamenon28. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
This is a 125,848,320-parameter base causal language model, trained from random weights on a packed English legal, financial, and educational corpus. It is a passage-completion model, not an instruction-following chat…
Downloads · 30 days
4
1% of all-time downloads
All-time downloads
552
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors503 MB · 100%
From the Hugging Face model README
This is a 125,848,320-parameter base causal language model, trained from random weights on a packed English legal, financial, and educational corpus. It is a passage-completion model, not an instruction-following chatbot, legal adviser, or factual authority.
| Metric | Value |
|---|---|
| Requested horizon | 2 epochs |
| Optimizer steps | 7,643 |
| Input tokens seen | 4,007,133,184 |
| Final validation loss | 2.2201357403 |
| Final validation perplexity | 9.2085807565 |
| Effective global batch | 524,288 input tokens |
| Hardware | Single node, 8× NVIDIA H100 |
| Precision | bfloat16 |
| Context length | 1,024 |
| Vocabulary | 16,384 |
| Actual production GPU charge | USD 23.06837 |
| Actual production total charge | USD 23.41963 |
Validation perplexity was computed over the complete validation split using token-weighted next-token cross-entropy across 20,210,388 predicted tokens.
(0.9, 0.95)0.1 (matrix parameters; biases and one-dimensional parameters excluded)6e-4; floor 6e-51.0The packed training split contains 2,003,556,352 tokens and the validation split contains 20,230,144 tokens. Every record is an already-packed 1,024-token uint16 window. The source mixture was built from:
HFforLegal/case-lawPleIAs/SECHuggingFaceFW/fineweb-edu (sample-10BT)The data pipeline applied document cleaning, exact/near deduplication, legal-evaluation decontamination, a custom 16,384-token tokenizer, EOS-delimited packing, and a deterministic 99:1 train/validation window split. Review each upstream dataset's terms before downstream commercial or regulated use; this repository does not relicense upstream data.
| Step | Validation loss | Perplexity |
|---|---|---|
| 1,000 | 2.7723 | 15.9950 |
| 2,000 | 2.5223 | 12.4573 |
| 3,000 | 2.4149 | 11.1890 |
| 4,000 | 2.3437 | 10.4192 |
| 5,000 | 2.2922 | 9.8963 |
| 6,000 | 2.2537 | 9.5232 |
| 7,000 | 2.2288 | 9.2887 |
| 7,643 | 2.2201 | 9.2086 |
Two one-H100 smoke runs preceded production. Run 1 reached validation but failed during metric-event logging; run 2 passed after that logging defect was fixed. The production run completed with no error events.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "lekhamenon28/legal-slm-125m-2ep"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "The court held that the contractual provision"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))