Downloads · 30 days
14
21% of all-time downloads
slm-125m-med-base
k2walia/slm-125m-med-base
slm-125m-med-base is a text generation model from k2walia. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 125.8M-parameter medical base language model, pretrained from scratch on a cleaned, deduplicated, and decontaminated ~1.76B-token medical corpus. This is a base (completion) model — it continues text, it is not inst…
All-time downloads
67
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors503 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Jul 18, 2026
- Updated
- Jul 18, 2026
- SHA
- 99100d07
Try a prompt
Model card
From the Hugging Face model README
slm-125m-med-base
A 125.8M-parameter medical base language model, pretrained from scratch on a cleaned, deduplicated, and decontaminated ~1.76B-token medical corpus. This is a base (completion) model — it continues text, it is not instruction-tuned or a chat model.
Model details
| Architecture | Llama (decoder-only) |
| Parameters | 125,848,320 (~125.8M) |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context length | 1024 |
| Vocab | 16,384 (byte-level BPE, trained on this corpus) |
| Tied embeddings | yes |
Training
| Tokens seen | 1.76B (1 epoch) |
| Optimizer steps | 3,360 |
| Hardware | 8× H100 (data-parallel) |
| Wall-clock | ~12 minutes |
| Compute cost | ~$6–7 |
| Validation perplexity | 27.26 |
Optimizer AdamW (betas 0.9/0.95, wd 0.1), cosine LR 6e-4→6e-5 with 200M-token warmup, global batch 524,288 tokens, seq len 1024, bf16.
Data mix (abstracts-first)
Built from public, ungated sources, streamed and cleaned from scratch:
| Source | Share | What |
|---|---|---|
| MedRAG/pubmed | ~82% | PubMed title+abstract snippets |
| HuggingFaceFW/fineweb-edu | ~12% | educational web (fluency) |
| epfl-llm/guidelines | ~5% | clinical practice guidelines |
| MedRAG/textbooks | ~1% | USMLE medical textbooks |
The corpus was decontaminated against PubMedQA, MedMCQA, and MedQA-USMLE (13-gram overlap removed) so downstream medical-QA evaluation is not leaked.
Intended use & limitations
Base model for research and as a starting point for fine-tuning (e.g. medical QA). It is not a medical device and must not be used for clinical decisions or to provide medical advice. Outputs may be inaccurate, outdated, or fabricated.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("k2walia/slm-125m-med-base")
model = AutoModelForCausalLM.from_pretrained("k2walia/slm-125m-med-base")
ids = tok("The patient presented with acute onset dyspnea and", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=64, temperature=0.8, do_sample=True)
print(tok.decode(out[0], skip_special_tokens=True))
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- apache-2.0
- Languages
- en
- Created
- Jul 18, 2026
- Updated
- Jul 18, 2026