tiny-llm-50m

Tiny-LLM-From-Scratch (~50M)

A ~50M parameter GPT-2-style decoder-only Transformer built entirely from scratch in PyTorch and pre-trained on FineWeb-Edu.

This model was created for educational purposes to demonstrate how a GPT-style LLM works at the tensor/code level. Every component—tokenizer, data pipeline, attention, MLP, training loop, and generation—is implemented from scratch without using the transformers library.

Architecture Details

Training Details

Sampling Capabilities

Supports autoregressive text generation with:

• Temperature scaling
• Top-K filtering
• Nucleus (Top-P) sampling

Usage (PyTorch)

from huggingface_hub import hf_hub_download
import torch, json

# Download files
config_path = hf_hub_download(repo_id="tiny-llm-50m", filename="config.json")
weights_path = hf_hub_download(repo_id="tiny-llm-50m", filename="pytorch_model.bin")

# Rebuild model (copy GPTConfig and GPT classes from the training notebook)
with open(config_path) as f:
    cfg_dict = json.load(f)
config = GPTConfig(**cfg_dict)
model  = GPT(config)
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

Source Code

Full project source: Tiny-LLM-From-Scratch