gpt2-small-fineweb-edu-10b

GPT-2 Small (124M) — reproduced on 10B tokens of FineWeb-Edu

A from-scratch reproduction of GPT-2 small (124M), trained on 10B tokens of FineWeb-Edu (sample-10BT) on a single H100 (~13 h). The custom training implementation was converted to a standard GPT2LMHeadModel for distribution (verified numerically exact vs. the original model, max logit diff ~1e-6).

• Code: https://github.com/submarat/gpt2-small-repro
• Write-up: https://submarat.github.io/reproducing-gpt2-small/

Training

Evaluation (0-shot, lm-evaluation-harness)

The profile is the FineWeb-Edu data fingerprint: it beats GPT-2 on knowledge (arc_easy, sciq), is near-parity on commonsense (hellaswag, piqa), and trails on distribution-sensitive tasks (lambada = narrative text, wikitext = encyclopedic) that FineWeb-Edu's quality filtering under-represents relative to GPT-2's WebText.

Usage

from transformers import GPT2LMHeadModel, GPT2TokenizerFast

model = GPT2LMHeadModel.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")
tok = GPT2TokenizerFast.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")

ids = tok("The process of photosynthesis", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_k=40, temperature=0.8)
print(tok.decode(out[0]))

Notes & caveats

• This is a 124M base model (~GPT-2-2019 level): no instruction following, it hallucinates, reasoning is weak. It is an educational / research artifact, not a product, and is not competitive with modern small models trained on far more data.
• Activation: trained with exact (erf) GELU, so the config sets activation_function="gelu" (not GPT-2's tanh-approx gelu_new).
• Intended uses: studying pretraining dynamics, interpretability (an induction head forms at layer 11 within the first ~2B tokens), and as a small research base model.