MiniGPT-Scratch-5M

MiniGPT 🚀

A GPT-style language model built completely from scratch in PyTorch for educational purposes.

This project implements the complete GPT pipeline, including tokenizer training, Transformer architecture, language model training, and text generation without relying on pre-built GPT implementations.

---

For installation check the repo:

https://github.com/samdoom-coder/MiniGPT.git

---

Features

• ✅ Custom BPE Tokenizer
• ✅ Token Embeddings
• ✅ Positional Embeddings
• ✅ Multi-Head Self-Attention
• ✅ Feed Forward Network (FFN)
• ✅ Residual Connections
• ✅ Layer Normalization
• ✅ Transformer Decoder Blocks
• ✅ Weight Tying (GPT-2 Style)
• ✅ Cross Entropy Loss
• ✅ AdamW Optimizer
• ✅ Temperature Sampling
• ✅ Top-k Sampling
• ✅ Top-p (Nucleus) Sampling
• ✅ EOS Stopping
• ✅ Model Checkpoint Saving & Loading

---

Model Architecture

---

Dataset

The model is trained on:

• TinyStories
• Approximately 1,000 stories (initial experiment)

Dataset:

https://huggingface.co/datasets/roneneldan/TinyStories

---

Tokenizer

A Byte Pair Encoding (BPE) tokenizer was trained from scratch using the TinyStories dataset.

Special tokens:

• <pad>
• <bos>
• <eos>
• <unk>

Vocabulary Size:

8000

---

Training

Optimizer

AdamW

Loss Function

CrossEntropyLoss

Learning Rate

3e-4

Batch Size

16

Epochs

20

---

Text Generation

MiniGPT supports multiple decoding strategies:

• Greedy Decoding
• Temperature Sampling
• Top-k Sampling
• Top-p (Nucleus) Sampling

Example:

output = model.generate(
    input_ids,
    max_new_tokens=200,
    temperature=0.8,
    top_k=100,
    top_p=0.9,
)

---

Example Output

Prompt

Once upon a time

Output

There was a little girl with dark hair and a smile.
She was very happy.
She ran to her mom and showed her the new dress.
Her mom smiled and said,
"Thank you, Lily.
You are a very good girl."

Lily smiled and said,
"I love you too, mom.
You are very kind."

---

Project Structure

MiniGPT
│
├── src/
│   ├── attention.py
│   ├── config.py
│   ├── dataset.py
│   ├── embeddings.py
│   ├── feedforward.py
│   ├── model.py
│   ├── trainer.py
│   └── transformer.py
│
├── tokenizer/
│   ├── tokenizer.json
│   └── tokenizer_config.json
│
├── train.py
├── generate.py
├── train_tokenizer.py
└── README.md

---

Future Improvements

• Larger model architecture
• More training data
• Validation dataset
• Learning rate scheduler
• Mixed precision (AMP)
• Flash Attention
• KV Cache for faster inference
• Hugging Face Transformers compatibility
• Model quantization
• Fine-tuning support

---

Purpose

This project was created to understand how GPT-style language models work internally by implementing every major component from scratch instead of using existing GPT implementations.

The goal is educational: to learn the architecture, training process, and text generation pipeline of modern decoder-only Transformer language models.

---

Tech Stack

• Python
• PyTorch
• Hugging Face Datasets
• Hugging Face Tokenizers
• Transformers

---

License

MIT License

---

⭐ If you found this project interesting or helpful, consider giving it a like!