Qwen3.5-4B-LumenGDN

Qwen3.5-4B-LumenGDN

Qwen/Qwen3.5-4B-Base with every attention layer replaced by a Gated DeltaNet. No attention anywhere, so no KV cache anywhere — a fixed-size associative memory and an O(1) update at every context length.

This model has not been healed.

The surgery is done; the training that follows it is not. The 24 layers the donor already trained as DeltaNets carry their own weights, but the 8 folded attention layers are transplants that have never taken a gradient step, and they lose rotary position in the fold. Expect degraded output. This is published as a starting point for that training and as a checkable record of the transplant — not as a model to deploy.

What was done

Qwen3.5-4B-Base is a 3:1 hybrid. Its layer_types reads lllA eight times over, so 24 of its 32 layers are already gated DeltaNets and only 8 are attention. Folding those 8 makes the whole model recurrent.

The 24 transplant exactly — verified to 2.7e-07 relative against Qwen's own layer in fp64. Four conventions differ between the two libraries and three of them are silent: a centred RMSNorm storing scale - 1, a 1/sqrt(d_k) query scale the output norm does not absorb, and a per-head decay scale outside the softplus. A fifth appears only above 0.8B, where the donor ties each q/k pair to several value heads: the tying is undone by a repeat_interleave one line before Qwen's kernel runs, so the scan is diagonal and the grouping is weight sharing on the projection alone. Modelling it as a grouped state would cost the per-state write gate, which this donor genuinely uses — the two beta directions sharing a key head sit at mean cosine 0.49 across 384 pairs, and the worst are anti-correlated.

The 8 are folded rather than deleted. Attention's q_proj (its query half), its gate half, k_proj/v_proj broadcast out of grouped-query attention, and o_proj become a DeltaNet keeping attention's own head geometry, with causal-identity short convolutions so the layer computes its transplanted projections exactly and earns its convolution by training. Each folded layer is calibrated against what the donor actually fed o_proj, and writes at half that amplitude.

The vision tower (333.5M) and the multi-token-prediction head (120.6M) are not included. This is the text path only, and that amputation is logit-identical to the donor's own text path at max |logit diff| = 0.000e+00.

The parameter count goes up: attention's tied and grouped projections are broadcast so every state owns its own query, key and value. 252M of the increase is exact duplicates at initialisation.

Loading it

This is not a transformers architecture; AutoModel will not load it. It needs Lumen for the layer, and modeling_qwen35_lumen.py in this repository for the assembly.

pip install lumen installs a different package. The lumen on PyPI is a data-visualisation library by other authors. The one this needs is the architecture library below, which is not published to PyPI.

pip install torch safetensors huggingface_hub
pip install git+https://github.com/latticedynamics/lumen.git

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("kennethgrace/Qwen3.5-4B-LumenGDN")
sys.path.insert(0, path)
from modeling_qwen35_lumen import load_creature

model, manifest = load_creature(path, device="cuda")
logits = model(input_ids)                     # parallel over the whole sequence
hidden, states = model.prefill(input_ids)     # or stream it
logits, states = model.step(next_id, states)  # O(1) per token, forever

qwen35.json carries the shape, the gate initialisation and the write-scale calibration the build used, so the model rebuilds itself without reaching for the donor's config. Weights are fp32 on purpose: error compounds with depth in a freshly collapsed stack, and bf16 is not safe here until roughly a thousand training steps in.

Credit

The weights are derivative of Qwen/Qwen3.5-4B-Base by the Qwen team, used and redistributed under Apache 2.0; the original LICENSE travels with this repository. The gated delta rule is Yang et al. This is an independent research artifact, not affiliated with or endorsed by Alibaba or the Qwen team.