gdn2-1.3B-fineweb-edu-100b

GDN-2 1.3B (FineWeb-Edu 100B)

A pure-recurrent linear-attention language model trained from scratch on FineWeb-Edu. Architecture: Gated DeltaNet-2 (GDN-2) — decoupling erase and write gates in linear attention.

A new checkpoint is uploaded roughly every 5B trained tokens (checkpoint-5B, checkpoint-10B, checkpoint-15B, …, checkpoint-100B).

---

Model configuration

name              = "gdn2_1.3B"
block_size        = 4096          # training context length
vocab_size        = 32000         # TinyLlama tokenizer
n_layer           = 18
n_head            = 18
n_embd            = 2304
head_dim          = 128
intermediate_size = 6208          # LLaMAMLP expansion
gdn2_per_layer    = 1             # 1 = pure recurrent, no SWA fallback
local_window      = 2048          # unused when gdn2_per_layer=1
rotary_percentage = 1.0
norm              = FusedRMSNorm (eps=1e-5)
mlp               = LLaMAMLP
parallel_residual = False
mamba_init        = True

---

Training recipe

Launch script: off/GatedDeltaNet-2/scripts/pretrain_gdn2_1.3B_fineweb_edu_100bt.sh

---

How to load

The checkpoint is a raw PyTorch state dict in the layout used by lit_gpt.model.GPT configured with gdn2_1.3B.

import torch
from lit_gpt.config import Config
from lit_gpt.model import GPT

ckpt = torch.load("checkpoint-15B-model-ckpt.pth", map_location="cpu", weights_only=False)
state = ckpt["model"] if "model" in ckpt else ckpt

cfg = Config.from_name("gdn2_1.3B")
model = GPT(cfg)
model.load_state_dict(state, strict=True)
model.eval()

The lit_gpt/ package used for training and inference lives at off/GatedDeltaNet-2/lit_gpt/ in the source repo.

---

Checkpoints

Each checkpoint-{N}B-model-ckpt.pth is a self-contained state dict (~17.4GB) at the {N}B-token training milestone. The latest is also mirrored as latest-model-ckpt.pth.

---

Intended use

Released for research purposes only.

Appropriate: studying GDN-2 recurrence, comparing linear/recurrent architectures (Mamba-2, Gated DeltaNet, KDA, RetNet), long-context retrieval experiments, component-level ablations.

Inappropriate: production deployment, safety-critical tasks, downstream benchmarks (wait for post-training evaluation on the final 100B checkpoint).

---

Limitations

• Mid-training. Loss is still decreasing; downstream metrics will move.
• No instruction tuning. Outputs are raw next-token completions.
• English-only training data (FineWeb-Edu is English academic web).
• 4K training context. Longer-context generalization not yet evaluated.