gpt-from-scratch-with-shakespeare

GPT From Scratch (Character Level)

This model is a character-level Transformer language model trained on the Shakespeare dataset. It was built from scratch using PyTorch, following Andrej Karpathy's "Let's build GPT" tutorial.

Training Steps: 2000 iterations

Model Architecture

• Embeddings: 384 dimensions
• Heads: 4 heads
• Layers: 4 transformer blocks
• Block Size: 256 characters context window
• Dropout: 0.2

Training results

How to Use

To use this model, you need to download the model.py file to define the architecture and load the weights.

import torch
import pickle
from model import LanguageModel # Imports the class from the uploaded model.py

# 1. Load Tokenizer Mappings
with open('tokenizer_meta.pkl', 'rb') as f:
    meta = pickle.load(f)
stoi, itos = meta['stoi'], meta['itos']
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join([itos[i] for i in l])

# 2. Initialize Model
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Note: Using the exact architecture from training
model = LanguageModel(
    vocab_size=len(meta['vocab_size']), # derived from meta
    n_embd=384,
    block_size=256,
    n_head=4, 
    n_layer=4
)

# 3. Load Weights
model.load_state_dict(torch.load('pytorch_model.pth', map_location=device))
model.to(device)
model.eval()

# 4. Generate Text
context = torch.zeros((1, 1), dtype=torch.long, device=device)
generated_ids = model.generate(context, max_new_tokens=500)[0].tolist()
print(decode(generated_ids))