Downloads · 30 days
15
4% of all-time downloads
arianraje/qwen3-4b-gdn-hybrid-init-dtfix
qwen3-4b-gdn-hybrid-init-dtfix is a text generation model from arianraje. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Surgery/init checkpoint (dtbias fixed) — uniform 1:4 GDN conversion of Qwen3-4B with plain weight inheritance, dtbias from the fla reference init (surgeryreport.json gdndecayinit; gdndecaystats.json = activation-level…
Downloads · 30 days
15
4% of all-time downloads
All-time downloads
341
Public
Parameters
4.5B
9.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.1 GB · 100%
From the Hugging Face model README
Surgery/init checkpoint (dt_bias fixed) — uniform 1:4 GDN conversion of Qwen3-4B with plain weight inheritance, dt_bias from the fla reference init (surgery_report.json gdn_decay_init; gdn_decay_stats.json = activation-level retention on wikitext: 0/864 dead heads vs 281/864 for the original series).
Part of a study converting full-attention Qwen3-4B into a GDN (gated
DeltaNet) hybrid (27 of 36 layers converted, uniform 1:4 retention) and
recovering capability via staged distillation. Checkpoints are stock
Qwen3NextForCausalLM — load with AutoModelForCausalLM (transformers >= 4.57).
-dtfix series. transformers' Qwen3Next port initializes the GDN time-step
bias with dt_bias.fill_(1.0); with A_log = log U(0,16) and the decay
g = -A * softplus(a + dt_bias) that gives retention exp(-dt*A) < 1e-3 for
2/3 of heads at init — the recurrent state starts dead and the gradient into
dt_bias/A_log is too small to recover it (the original qwen3-4b-gdn-hybrid-*
series carries this through every stage). This series re-runs the pipeline from
a surgery that uses the fla GatedDeltaNet reference init
(dt ~ LogUniform(1e-3, 0.1), dt_bias = softplus^-1(dt); median retention 0.94).
Everything else (inheritance map, data recipe, trainer, LRs, budgets) is unchanged.