slm-from-scratch-phase3-scaling

slm-from-scratch-phase3-scaling

Three GPT-style language models, hand-written and trained from scratch in PyTorch on a single RTX 5070 Ti, produced by a controlled scaling-law experiment: same architecture family, varying only model size and tokens seen. No HuggingFace transformers model classes were used — every component (attention, MLP, blocks, embeddings, training loop) was hand-coded.

This is Phase 3 of a larger build documented at github.com/Ishaanred/slm-from-scratch. See docs/phase3/results.md for the full scaling-law writeup and plots. These are portfolio and learning artifacts, not production models.

The three checkpoints

Each subfolder is independently loadable: model.safetensors + config.json. All three share the same model.py and tokenizer (this repo's root).

Note: these are the best validation-loss checkpoints saved during each run (the lowest point reached), not necessarily the literal last training step — the scaling-law docs report final-iteration loss instead, for a compute-matched comparison across runs. The two numbers differ slightly because validation loss is noisy near the end of a cosine-decay schedule.

What they're for

• 75m-500m-tokens vs 75m-2b-tokens — same model size, 4x the tokens. Tests whether more data helps at fixed size.
• 75m-500m-tokens vs 150m-500m-tokens — same tokens, ~1.9x the parameters. Tests whether more parameters help at fixed, scarce data.

Architecture

Training

Honest assessment

These are small models trained on a token budget far below Chinchilla-optimal for their size (compute-optimal would be roughly 20x tokens/param; the largest run here reaches ~29x on the small model, but only ~3.7x on the 150M model). They generate grammatical English with local coherence but no long-range consistency, and score near the random-chance floor on standard benchmarks (see docs/phase5/evaluation.html) — expected at this scale, not a bug.

Files

• <checkpoint-name>/model.safetensors — model weights, fp32, optimizer state stripped
• <checkpoint-name>/config.json — architecture and training metadata for that checkpoint
• model.py — the from-scratch model definition needed to load any of these weights
• tokenizer/ — the project's own 32K BPE tokenizer (tokenizer.json, vocab.json, merges.txt)

Usage

This is not a transformers architecture, so load it with the included model.py.

import json, torch
from safetensors.torch import load_file
from model import GPT, GPTConfig
from tokenizers import Tokenizer

CHECKPOINT = "75m-2b-tokens"  # or "75m-500m-tokens" / "150m-500m-tokens"

cfg = json.load(open(f"{CHECKPOINT}/config.json"))
model = GPT(GPTConfig(
    n_layer=cfg["n_layer"], n_head=cfg["n_head"], n_embd=cfg["n_embd"],
    block_size=cfg["block_size"], vocab_size=cfg["vocab_size"], dropout=0.0,
))
model.load_state_dict(load_file(f"{CHECKPOINT}/model.safetensors"))
model.eval()

tok = Tokenizer.from_file("tokenizer/tokenizer.json")
ids = torch.tensor([tok.encode("The meaning of life is").ids])

with torch.no_grad():
    for _ in range(50):
        logits, _ = model(ids[:, -cfg["block_size"]:])
        nxt = torch.softmax(logits[:, -1, :] / 0.8, dim=-1).multinomial(1)
        ids = torch.cat([ids, nxt], dim=1)
print(tok.decode(ids[0].tolist()))

Limitations

• Small and undertrained relative to Chinchilla-optimal, especially the 150M checkpoint.
• Trained on OpenWebText, so it inherits the biases and noise of unfiltered web text scraped from Reddit-linked URLs.
• No instruction tuning, no safety alignment. Raw next-token predictors.

License

MIT. Built as an educational project.