hybrid-gpt-232m

hybrid-gpt-232m

A 232M-parameter research base model with hybrid local+global attention (sliding window w=64 on 12 of 16 layers, full causal attention every 4th layer), trained entirely on one Apple M4 Pro Mac mini on 1.0B tokens of refined data. This card is self-contained — the training code lives in a research repo (github.com/GarageLM/garagelm) that may be private; everything needed to understand and evaluate the model is below.

Why it's interesting:

• The hybrid attention pattern beat full attention at equal parameters in controlled experiments (3/3 random seeds, mean gap −0.038 nats), at ~30% of full attention's KV cache (measured: 4.97MB at 1024 context).
• At matched, locally re-run evaluation it is ahead-or-tied vs Pythia-160M (300B training tokens) on all four tasks below with 300x fewer tokens (within n=300 eval resolution), and ahead of gpt2 on ARC-Easy — the one gap beyond 2× the slice standard error.

(0-shot acc/acc_norm via lm-evaluation-harness, n=300/task, identical harness and example slices for every row — the reference models were re-run locally rather than quoting published full-set numbers, which use different settings and are not comparable.)

Architecture

Decoder-only transformer: 16 layers, d_model 1024, 16 heads / 4 KV heads (GQA), SwiGLU FFN (hidden 2816), RoPE (θ=10000), RMSNorm, weight tying, context 1024, gpt2 BPE (vocab 50257). Attention is a sliding window of 64 tokens except layers 3/7/11/15, which are fully causal — the KV cache at full context is ~30% of an all-global equivalent, decoding at ~310 tok/s fp16 / ~530 tok/s 4-bit in MLX on M4 Pro (TTFT ≤75ms).

Training

• Data: 1.0B tokens sampled from a 3.55B-token pool — 84.5% FineWeb-Edu (sample-10BT shards) + 15.5% Cosmopedia-v2 synthetic textbooks. Sub-epoch (each token seen <1x on average).
• Recipe: AdamW, cosine LR 3e-4→3e-5 (warmup 200), weight decay 0.1, grad clip 1.0, 61,000 steps × 16,384 tokens/step, dropout 0.0, seed 1337. ~127h wall-clock on a single M4 Pro (48GB, PyTorch/MPS), fp32.
• Validation: final val loss 3.070 (PPL 21.5) on a 2M-token held-out split of the training distribution.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("garagelm/hybrid-gpt-232m")
model = AutoModelForCausalLM.from_pretrained("garagelm/hybrid-gpt-232m", trust_remote_code=True)
out = model.generate(**tok("The water cycle begins when", return_tensors="pt"), max_new_tokens=100)
print(tok.decode(out[0]))

No KV cache in this wrapper (research release; generation re-runs the prefix each step).

Apple Silicon (MLX)

The MLX conversions are the runtime the throughput numbers above were measured under, and unlike this wrapper they carry real per-layer KV caches (bounded on the 12 sliding-window layers):

• hybrid-gpt-232m-mlx — float16, 464MB
• hybrid-gpt-232m-mlx-4bit — 4-bit, 130MB, at +0.018 nats validation loss

Both are logit-parity-gated against this PyTorch implementation.

Limitations

• A 232M-parameter research artifact, not an assistant. Fluent text, unreliable facts, 1024-token context, English only, no alignment or safety tuning of any kind.
• Trained on FineWeb-Edu and Cosmopedia-v2 (both ODC-BY); inherits their biases and educational-web distribution.
• MMLU-class benchmarks are at chance, as for all models this size.
• Milestone 13's sparse arm, garagelm/hybrid-gpt-moe-284m-a114m (8 experts, top-2, 284M total / 114M active, 100M tokens), is released as a reproducibility artifact with its active-matched and memory-matched controls on its card; it is not a stronger model than this one.
• The chat-tuned variant is garagelm/hybrid-gpt-232m-chat (v2: 310M-token SmolTalk SFT + UltraFeedback DPO; IFEval 13.9% prompt-strict, at a disclosed ~2-point full-set ARC-Easy cost vs this base — details on its card).