cyberslm-33m-base

CyberSLM-33M-Base

A 33.5M-parameter cybersecurity-focused small language model, pretrained from scratch on a ~745 MB curated corpus (cybersecurity, computer fundamentals, programming, general English) with a custom 32k SentencePiece BPE tokenizer. Trained on a single NVIDIA L40S.

This is the base (pretrained-only) model. For the instruction-tuned variant see cyberslm-33m-instruct.

Architecture

Decoder-only transformer (Llama-style, loadable with LlamaForCausalLM):

Training

• Corpus: ~665k documents (~205M tokens), 95/5 train/val split, seed 42
• 4,000 steps, effective batch 131,072 tokens/step (~524M tokens seen)
• AdamW β=(0.9, 0.95), wd 0.1, cosine schedule, peak LR 3e-4, grad clip 1.0
• Final val loss: 2.43

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("sabari2005/cyberslm-33m-base")
model = AutoModelForCausalLM.from_pretrained("sabari2005/cyberslm-33m-base")

ids = tok("A SQL injection attack is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Special tokens: pad=0 (<pad>), unk=1 (<unk>), bos=2 (<bos>), eos=3 (<eos>).

Limitations

Base model — no instruction tuning; it continues text rather than follows instructions. Trained on ~0.5B tokens, so expect limited general knowledge outside the cyber/CS domain. English only.