shakespeare-gpt

Shakespeare GPT

A small character-level GPT language model (~10.8M parameters) trained from scratch on the Tiny Shakespeare dataset following Andrej Karpathy's "Let's build GPT" tutorial.

Model Details

• Architecture: decoder-only Transformer
• Vocab size: 65 (character-level)
• Embedding dim: 384
• Number of heads: 6
• Number of layers: 6
• Context length (block size): 256
• Total parameters: 10.79M

Training

• Dataset: Tiny Shakespeare
• Optimizer: AdamW (lr=0.001)
• Iterations: 5000
• Final train loss: 0.8415
• Final val loss: 1.5810

Usage

This is a custom PyTorch model. See model.py for the architecture, config.json for hyperparameters, and vocab.json for the character-level tokenizer mappings.

import json, torch
from huggingface_hub import hf_hub_download
from model import GPTLanguageModel


config = json.load(open(hf_hub_download('achavan1211/shakespeare-gpt', 'config.json')))
vocab = json.load(open(hf_hub_download('achavan1211/shakespeare-gpt', 'vocab.json')))
stoi, itos = vocab['stoi'], {int(k): v for k, v in vocab['itos'].items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join(itos[i] for i in l)

model = GPTLanguageModel(**config)
model.load_state_dict(torch.load(hf_hub_download('achavan1211/shakespeare-gpt', 'pytorch_model.bin'), map_location='cpu'))
model.eval()

# Generate text
context = torch.zeros((1, 1), dtype=torch.long)
print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))