gpt-124m-param-600m-tokens

GPT Language Model

A 500M parameter GPT model trained from scratch using PyTorch.

This project contains:

• custom multi-head self-attention
• transformer blocks
• causal masking
• autoregressive text generation
• mixed precision training
• top-k / top-p sampling
• safetensors model weights

The model was trained on a subset of FineWeb-Edu using a GPT-2 tokenizer.

---

Architecture

Model configuration:

{
    "vocab_size": 50257,
    "context_length": 256,
    "emb_dim": 768,
    "n_heads": 12,
    "n_layers": 12,
    "drop_rate": 0.1,
    "qkv_bias": False
}

Approximate parameter count:

• ~500M parameters

Architecture components:

• token embeddings
• positional embeddings
• masked multi-head self-attention
• feed-forward MLP blocks
• pre-layer normalization
• residual connections
• causal language modeling head

---

Training

Training setup:

• PyTorch
• AdamW optimizer
• Automatic Mixed Precision (AMP)
• Gradient clipping
• Top-k / Top-p text generation

Hardware used:

• RTX 4090 24GB VRAM

Dataset:

• FineWeb-Edu subset (Continued from a 100M token checkpoint, trained on an additional 500M tokens for a total of 600M tokens)

Tokenizer:

• GPT-2 tokenizer

---

Training Progress

The graph below shows train/validation loss progression during training on FineWeb-Edu.

Installation

Install dependencies:

pip install torch transformers safetensors

---

Loading The Model

import json
import torch

from safetensors.torch import load_file
from transformers import AutoTokenizer

from model import GPTModel

# load config
with open("config.json") as f:
    cfg = json.load(f)

# create model
model = GPTModel(cfg)

# load weights
state_dict = load_file("model.safetensors")

model.load_state_dict(state_dict)

model.eval()

# tokenizer
tokenizer = AutoTokenizer.from_pretrained(".")

---

Text Generation Example

from model import generate_and_print_sample

print(generate_and_print_sample(model, tokenizer, "cuda", "The world is big"))

---

Sample Generations

Example generations from early-stage training:

"The world is big and powerful. “It will be a huge step forward in the right direction,” said John Kopp, CEO of Google Earth. “This will help us understand what we are looking for now to come, how it will affect our health and well-being.”<|endoftext|>"

The model currently demonstrates

• Strong generation of academic, scientific, and medical prose
• Coherent paragraph structure and article-like formatting
• Proper syntax for quotes, attributions, and citations
• Generation of structured outputs like bulleted lists
• Stable grammar and vocabulary diversity in narrative contexts
• Ability to hallucinate plausible-sounding names, institutions, and journals

Current limitations:

• Severe factual inaccuracies and completely hallucinated facts
• Prone to getting stuck in repetitive loops (especially when generating lists or enumerations)
• Semantic drift and hollow sentences (sentences are grammatically correct but often lack underlying logic)
• Limited reasoning depth
• No instruction following capabilities (it is a base autoregressive model)

---

Files

model.py              # GPT architecture
model.safetensors     # trained weights
config.json           # model configuration
tokenizer files       # GPT2 tokenizer assets
README.md             # project documentation

---

Notes

This is a custom PyTorch implementation and is not directly compatible with Hugging Face AutoModelForCausalLM.

Users should load the model using the provided model.py architecture.

---

License

MIT License.

license: mit