gpt2

Run GPT-2 with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/gpt2

Paper: Language Models are Unsupervised Multitask Learners (Radford et al., 2019)

GPT-2 is OpenAI's decoder-only transformer language model trained on WebText: learned absolute position embeddings, pre-LayerNorm blocks, gelu_new activations, a tied output head, and a byte-level BPE tokenizer. This is the 124M variant, a base completion model (no chat template).

For more details, see the upstream model card.

Pure-Keras 3 conversion of openai-community/gpt2 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from zeromodels.models.gpt2 import GPT2TextGenerate, GPT2Tokenizer

model = GPT2TextGenerate.from_weights("zeromodels/gpt2")
tokenizer = GPT2Tokenizer.from_weights("zeromodels/gpt2")

inputs = tokenizer("The meaning of life is")
outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0]))

All GPT-2 sizes load the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• This is a base completion model: it continues a prompt and is not instruction-tuned.
• See the GPT-2 docs and Loading Weights.
• Upstream safetensors still work via the hf: prefix, e.g. GPT2TextGenerate.from_weights("hf:openai-community/gpt2").

Special Thanks

A huge thank you to the OpenAI GPT-2 authors for creating and releasing these models.

License: MIT, inherited from the upstream OpenAI GPT-2 release.