gpt2-50m

GPT-2 50M

Custom-trained GPT-2 checkpoint with deliberate depth-width configuration for inference benchmarking research.

Created as part of the Banterhearts research program investigating benchmarking integrity for local LLM inference.

Purpose

Mid-scale MHA model for scaling-regime validation.

These checkpoints are not general-purpose language models. They are deliberately sized scaling-study artifacts designed to isolate the effect of model depth vs width on GPU inference latency. The key finding: in the small-model GPU regime, layer depth (not parameter count) dominates latency, producing inversions where a 5M-parameter model can be 3.6x slower than a 25M-parameter model.

Source Technical Reports

Used in: TR117, TR126, TR147

Design Rationale

The GPT-2 family (25M, 50M, 100M) uses a 2x3 factorial design:

All models use 2 attention heads (MHA, not GQA) to isolate architecture effects from attention-group structure. Dropout is set to 0.0 for deterministic inference measurement.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Crusadersk/gpt2-50m")
tokenizer = AutoTokenizer.from_pretrained("Crusadersk/gpt2-50m")

inputs = tokenizer("Hello", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Compatibility

Citation

@misc{banterhearts2026gpt250m,
  title = {Custom GPT-2 Scaling Checkpoint (50M) for Inference Benchmarking Research},
  author = {Kadadekar, Sahil},
  year = {2026},
  url = {https://huggingface.co/Crusadersk/gpt2-50m},
  note = {Part of the Banterhearts research program. NeurIPS 2026 submission.}
}

Acknowledgments

This work is part of the Chimera/Banterhearts technical-report program on deployment-time LLM behavior, quantization, refusal robustness, batching effects, and inference-stack reliability. Canonical public archive: Chimeraforge Reports; source context: github.com/Sahil170595/Banterhearts.