transformer-from-scratch

transformer-from-scratch — trained checkpoints

Weights for github.com/adimunot21/transformer-from-scratch, a decoder-only Transformer built from first principles in PyTorch — attention, multi-head projections, positional encoding, blocks and training loop written by hand, plus a BPE tokenizer also implemented from scratch.

Two models, both trained 5000 steps on Tiny Shakespeare:

Each folder holds model.safetensors and a config.json with the exact architecture and training hyperparameters.

Loading

Plain state_dicts for the model class in the source repo:

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("adimunot/transformer-from-scratch", "bpe/model.safetensors")
model.load_state_dict(load_file(path))   # model built from config.json
model.eval()

Notes

• The optimizer state present in the original training checkpoints has been stripped — these are inference weights only. The char model went from 11.3 MB to 4.4 MB as a result.
• Intermediate step checkpoints (1000/2000/3000/4000) were not published; only the final step-5000 weights are here.