wcb-gpt2-124m-from-scratch

GPT-2 124M (from scratch) fine-tuned on World Central Banks

Fine-tuned classifiers for the three tasks of the World Central Banks (WCB) benchmark from Words That Unite The World (NeurIPS 2025 Datasets & Benchmarks):

The backbone is a GPT-2 124M implemented from scratch (no transformers modelling code) and pretrained from random init on FineWeb-Edu. LoRA and QLoRA (blockwise NF4 with double-quantized scales) are also hand-written -- no peft, no bitsandbytes.

Checkpoints

Metrics are on the WCB test split for seed config 5768. The published checkpoint is that seed; the numbers reported in the paper-comparison tables below are means over all three of the paper's seed configs (5768, 78516, 944601).

Base model: base/pretrain_fineweb_edu_9.9B.pt

GPT-2 124M written from scratch and pretrained from random init on one epoch of FineWeb-Edu's 9.9B-token sample, 3x RTX 6000 Ada with DistributedDataParallel, 10.89 h wall.

Beating OpenAI's checkpoint on val loss is partly home-turf advantage: the loss is measured on FineWeb-Edu's own validation split, and FineWeb-Edu is a cleaner corpus than the WebText GPT-2 trained on. HellaSwag is the fairer comparison and the margin there is small (+0.008).

The LoRA and QLoRA checkpoints require this file -- they contain only adapters and a classifier head.

Loading

These are not transformers checkpoints -- AutoModel will not load them. The repo ships the model code:

pip install torch tiktoken
python load_example.py stance lora

model.py (GPT-2 + classifier head), peft.py (NF4 + LoRA) and metrics.py are included, and load_example.py shows the full path for each mode.

The classifier reads the hidden state of the last real token. Padding is on the right and attention is causal, so pad positions cannot influence that vector and no attention mask is needed.

Results in context

Weighted F1, mean +- std over the paper's three seeds:

Pretraining is worth +0.067 weighted F1 on stance and much less on the other two tasks, which are largely cued by surface lexis that 17.5k labelled examples teach directly. The reproduced BERT/RoBERTa baselines land within 0.006 of the paper's published numbers on identical splits, which is the evidence that these GPT-2 rows are measured on the same footing.

LoRA trains 0.36% of the weights and still matches full fine-tuning on temporal and beats it on uncertainty -- with 17.5k examples against 124M parameters, freezing the backbone regularizes.

Full evaluation record, including per-bank breakdowns and per-epoch histories for every run, is in results/.

Limitations

• 124M parameters: below the paper's fine-tuned encoder PLMs on stance (0.668 vs 0.740 for RoBERTa-Large).
• Near-useless zero-shot (0.218 calibrated on stance). These are fine-tuned classifiers, not instruction-followers.
• Trained on English-language central bank communications, 1996-2024. Stance labels encode the annotation guidelines of the source paper; do not treat outputs as financial advice.

Citation

@inproceedings{shah2025words,
  title     = {Words That Unite The World: A Unified Framework for Deciphering
               Central Bank Communications Globally},
  author    = {Shah, Agam and Sukhani, Siddhant and Pardawala, Huzaifa and others},
  booktitle = {NeurIPS 2025 Datasets and Benchmarks Track},
  year      = {2025}
}

The WCB dataset is released under CC-BY-NC-SA 4.0; these derived models carry the same terms.