GPT2

GPT-2 Weights Repository

This repository contains the weights for my custom implementation of GPT-2.
For the codebase, please refer to the main repository: GitHub Link

Compatibility Note

My implementation has a different architecture structure than the original OpenAI release, but it aligns with the Hugging Face format available here:
Hugging Face GPT-2 (openai-community)

To ensure compatibility, I’ve included both the Hugging Face-compatible weights and the weights from my own training.

Files Included

• HF_Weights.pth
  Pretrained Hugging Face GPT-2 model weights adapted to work with my implementation.
  See this script for loading details:
  load_gpt2_weights.py

• gpt2.pth
  My custom-trained GPT-2 model, trained from scratch on the FineWebEdu-10BT dataset (trained on 40BT).

How to Load the Model

To use the weights with your GPT-2 implementation, follow the example below. The model is defined here model.py as GPT2. Below are two examples demonstrating how to load the model weights depending on whether you are using my GPT-2 or the Hugging Face-compatible weights.

Load My GPT-2

import torch
from model import GPT2

model_config = {
    "n_blocks": 12,
    "seq_len": 1024,
    "n_embd": 768,
    "n_head": 12,
    "vocab_size": 50304,
    "dropout": 0.1 # Used for training
}

model = GPT2(**model_config)
ckpt_path = 'gpt2.pth'
state_dict = torch.load(ckpt_path, map_location='cpu')["model_state_dict"]
model.load_state_dict(state_dict)
model.eval()

Load Hugging Face-Compatible Weights

import torch
from model import GPT2

hf_model_config = {
    "n_blocks": 12,
    "seq_len": 1024,
    "n_embd": 768,
    "n_head": 12,
    "vocab_size": 50257,
    "dropout": 0.0
}

model = GPT2(**hf_model_config)
ckpt_path = 'HF_Weights.pth'
state_dict = torch.load(ckpt_path, map_location='cpu')["model_state_dict"]
model.load_state_dict(state_dict)
model.eval()