SmolLM2-135M-LumenGDN

SmolLM2-135M collapsed into a Gated DeltaNet

This is SmolLM2-135M with its tensors converted and stitched into a Gated DeltaNet configuration: 30 layers, d_model=576, 3 states of width d_k=192 / d_v=512, one mixer per layer, no attention and no separate feed-forward anywhere in the model.

It is completely untrained in this form. The weights are the donor's, rearranged. Nothing here has taken a gradient step since the surgery.

What was done

Each donor decoder layer is two sub-layers — attention, then a gated feed-forward. Both fold into a single Gated DeltaNet. Five of the donor's seven matrices transfer with no slicing and no padding:

A Gated DeltaNet's output path already is an up/gate/down arrangement — the same shape as a SwiGLU — so the feed-forward has somewhere to land, while the query and key projections are already shaped for a key-addressed memory.

Embeddings, the tie, and every RMSNorm come across untouched. Attention's v_proj and o_proj are homeless — the feed-forward claimed the value and output side first — and are left out. The short convolutions are initialised to a causal identity (all taps zero but the last), so a freshly collapsed layer computes its transplanted projections untouched; the default convolution init would scramble them before a single gradient step.

The head partition does not survive: nine query heads of width 64 become three states of width 192. Identical numbers in identical positions, read against a different grouping. n_heads must divide the query rows and the feed-forward width, which leaves {3, 6, 12, 24, 48, 96, 192} — the donor's own 9 is not on the list, because 9 does not divide 1536.

Contents

• model.safetensors — 121,553,148 parameters, float32
• collapse.json — the manifest the loader reads (donor revision, layer count, plan)
• modeling_collapsed.py — standalone loader, depends only on torch + lumen
• tokenizer files — the donor's, unmodified

Use

pip install torch safetensors
pip install git+https://github.com/latticedynamics/lumen.git

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("kennethgrace/SmolLM2-135M-LumenGDN")
sys.path.insert(0, path)

from modeling_collapsed import load_collapsed

model, manifest = load_collapsed(path, device="cuda")
logits = model(ids)          # (B, T) token ids -> (B, T, vocab)

This is not a transformers architecture and will not load with AutoModelForCausalLM. Generation runs in constant memory through model.step(ids, states) rather than re-reading a growing prefix.

Provenance and license

Donor: HuggingFaceTB/SmolLM2-135M at revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2, Apache-2.0. This derivative is released under the same license. The Gated DeltaNet implementation is lumen, MIT.