tiny-llm-27m

tiny-llm — a 27M-parameter LLM built from scratch, end to end

A complete modern LLM pipeline built by hand as a learning mission — every stage written from scratch in readable PyTorch (no HF Transformers, no external tokenizer), trained on free Colab T4 and a Mac CPU.

Pipeline: own byte-level BPE tokenizer → pretraining → SFT (instruction following) → DPO (preference alignment) → distilled draft model + speculative decoding → RAG with copy-tuning → this release.

Sizes

Training data

• Only dataset: TinyStories (synthetic children's stories, English). The validation file was deduplicated against train (33% of it duplicated train stories) before any evaluation.
• SFT pairs: "Write a story about X.\n\n" + story, where X is strictly the story's first-sentence subject appearing ≥2 times (data-quality lesson: a noisier heuristic taught the model to ignore instructions).
• DPO pairs: chosen = human story, rejected = the SFT model's own sample.

Honest limitations

• Writes only simple children's stories in English. Nothing else.
• It does not know facts about the world, cannot do math, cannot chat in Russian or any other language.
• Instruction following ("about X") works on ~7/10 held-out topics; very rare words (e.g. "unicorn") may be ignored.
• The RAG variant only quotes from a supplied context line; without context it invents.
• Trained on ~500M tokens of synthetic text at 27M params — a toy, roughly 25,000× smaller than frontier models. Built for learning, not for use.

Usage

import torch
from model import TinyLLM, ModelConfig      # model.py from this repo
from bpe import BPETokenizer                # bpe.py from this repo

tok = BPETokenizer.load("tokenizer.json")
ck = torch.load("dpo-m4c.pt", map_location="cpu")
m = TinyLLM(ModelConfig(**ck["cfg"])); m.load_state_dict(ck["model"]); m.eval()

# IMPORTANT: encode word-by-word (the same word-cache encoding used in training)
import re
ids = []
for w in re.findall(r"\S+\s*", "Write a story about dragon.\n\n"):
    ids += tok.encode(w)
out = m.generate(torch.tensor([ids]), 120, temperature=0.7)
print(tok.decode(out[0].tolist()))

Speculative decoding (speculative.py) makes greedy generation ~1.4× faster on CPU with byte-identical output; rag.py + rag-corpus.txt show the RAG setup.

Validation

Every stage was gated by a frozen validation contract (18 assertions): pretrain val loss 1.656 (≤2.3), instruction following judged blind 7/10, DPO preferred blind in 90% of 20 pairs, speculative decoding exact on 20/20 prompts at 1.40×, RAG answers contain the fact on 9/10 questions (0/10 without retrieval). Fresh-context judges never saw the training code.