support-125M-slm-base

Support 125M SLM - Base

A 125M parameter Llama-style language model trained from scratch on ~2.6B tokens of curated IT support and technical data. This is the base (pretrained) model — it completes text but does not follow instructions.

Training Data

Total: ~2.85B tokens (6 epochs = ~17B tokens seen)

Model Architecture

Training Details

• Hardware: 8x H100 (Modal cloud)
• Framework: PyTorch + DDP
• Optimizer: AdamW (lr=6e-4, warmup 200M tokens, cosine decay)
• Mixed precision: bfloat16
• Total cost: ~$31
• Val perplexity: 15.06

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("applegrew/support-125M-slm-base")
tokenizer = AutoTokenizer.from_pretrained("applegrew/support-125M-slm-base")

prompt = "The VPN connection keeps dropping"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Special Tokens

This model uses custom chat tokens: <|bos|>, <|eos|>, <|pad|>, <|unk|>, <|system|>, <|user|>, <|assistant|>

SFT Version

For instruction following, use the SFT version: applegrew/support-125M-slm-sft