slm-pipeline-151m

slm-pipeline-151m

A 151M-parameter language model trained from scratch through every stage of the modern pipeline — pre-training → mid-training → SFT → DPO → RLVR/GRPO — on one H100 in 20 hours 20 minutes. Every stage is plain PyTorch: no TRL, no Trainer, every loss written out.

This is a demonstration of a training pipeline, not a useful assistant. Read the evaluation section before doing anything with it.

⚠️ This is not a transformers model

config.json declares "architecture": "slm-pipeline/Transformer". It is not a transformers architecture and AutoModelForCausalLM.from_pretrained will not load it. The weights are plain safetensors for the custom model defined in slm/model.py of the producing repo.

from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import torch, json

path = snapshot_download("alpharomercoma/slm-pipeline-151m")
cfg = json.load(open(f"{path}/config.json"))
state = load_file(f"{path}/model.safetensors")

# Requires slm-pipeline on your PYTHONPATH:
from slm.model import ModelConfig, Transformer
model = Transformer(ModelConfig.from_dict(cfg))
model.load_state_dict(state)
model.eval()

The tokenizer is a byte-level BPE trained on this corpus and loads with tokenizers.Tokenizer.from_file("tokenizer.json"). Chat turns use single reserved control tokens rather than ChatML text: <|bos|> <|user|> … <|end|> <|assistant|> … <|end|> <|eos|>.

Evaluation — read this before using it

Measured on the final 6_rlvr_grpo checkpoint, with the chat template.

MMLU is at chance and GSM8K is at the floor. HellaSwag and ARC-Easy are meaningfully above random, so the model has real commonsense signal — but at 151M parameters it has very little knowledge, cannot do arithmetic, and hallucinates freely. It should not be used for anything consequential.

Capability across the whole pipeline (single run, one seed, aggregate scores only — every multiple-choice movement between stages is inside one standard error, so read these as "nothing was lost", not as stage-by-stage gains):

The perplexity rise at SFT is not a regression — it is measured on held-out web text while the model is being specialised for dialogue.

The one large, unambiguous signal is the format-rate column, and it tells a three-stage story: mid-training taught the GSM8K #### <answer> convention (0.900), SFT destroyed it (0.055, because SmolTalk contains #### in 2,733 of 455,483 conversations), and GRPO recovered it (0.985). GRPO recovered a behaviour rather than discovering one, and it did not improve arithmetic: group-relative advantages give no gradient once every rollout in a group scores identically, which is what happens when all of them are well-formed and none correct.

Architecture

Training

Pre-training consumed 8.26B tokens from a 9.89B-token corpus — 0.84 epochs, so no token was seen twice. That is 54.7 tokens per parameter, deliberately far past Chinchilla-optimal (~3B), which is the right trade when you care about quality at a fixed inference cost.

Pre-training and mid-training are two halves of one Warmup–Stable–Decay schedule: stage 2 holds the LR flat at 6.0e-4 and ends hot; stage 3 picks it up there and anneals to exactly 0.0 on a better mixture. Peak MFU 0.412; pre-training held 317,600 tok/s at MFU 0.364 in steady state.

GSM8K train/test splits are disjoint by construction: gsm8k_text (mid-training) and the GRPO rollout prompts both come from split="train" (7,473 rows); evaluation uses split="test" (1,319 rows). No further decontamination against the web corpora was done.

Provenance

• Producing code: slm-pipeline at commit c71c1fe
• All 28 source files verified identical between the training host and local HEAD by md5, mid-run
• Hardware: 1× NVIDIA H100 80GB SXM, Nebius eu-north1, driver 550.163.01, torch 2.13.0+cu129
• evals/ in this repo contains the raw per-stage evaluation JSON and the generated progression table

Limitations

One run, one seed, one checkpoint per stage, scored once. No replicates, no confidence intervals on differences, and the evaluation harness stored aggregate accuracies only — so no paired testing is possible after the fact. Treat every between-stage difference here as un-established in either direction.

License

Apache-2.0 for the weights. Training data carries its own licences — FineWeb-Edu (ODC-By), FineMath, smol-smoltalk, UltraFeedback, GSM8K (MIT), MMLU auxiliary train — and you are responsible for complying with those if you build on this.