minillm-tinystories

MiniLLM: A Modern Transformer Language Model Built From Scratch

🚀 Try it live in a Space

A small decoder-only language model trained from scratch on the TinyStories dataset. Built as a hands-on learning project implementing modern architectural techniques (RoPE, GQA, SwiGLU, RMSNorm) from first principles.

Code + training script: GitHub repo

Model Details

Architecture

• RMSNorm (pre-norm)
• Rotary Positional Embeddings (RoPE)
• Grouped Query Attention (GQA)
• SwiGLU feed-forward network
• Weight-tied embedding/output layers

Usage

This model requires the custom MiniLLM architecture class to load (not a standard transformers architecture). See the GitHub repo for the full model code and a ready-to-run inference.py.

git clone https://github.com/IbrahimKhan7208/minillm-tinystories
cd minillm-tinystories
pip install -r requirements.txt
# download minillm_final.pt and tokenizer.json from this Hugging Face repo into weights/
python inference.py --prompt "Once upon a time" --temperature 0.8

Sample Output

Prompt: "Once upon a time"

Once upon a time, there was a little girl named Lily. She loved to play outside and explore. One day, she went on a walk and saw a big bright sun in the sky...

Intended Use & Limitations

Trained on a small, simple-vocabulary children's-story dataset for educational purposes. Not suitable for general-purpose or production text generation. No safety filtering has been applied — treat generations accordingly for a children's-story-style dataset.

Training Details

Optimizer: AdamW, LR 3e-4 (constant), gradient clipping at 1.0, batch size 64, ~13,000 steps (≈1 epoch over 1M TinyStories). Trained on a single Colab T4 GPU, ~3.5-4 hours total (run across multiple sessions using manual checkpoint save/resume).