Downloads · 30 days
20
23% of all-time downloads
arianraje/qwen3-4b-gdn-hybrid-stage2b-kd
qwen3-4b-gdn-hybrid-stage2b-kd is a text generation model from arianraje. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Stage-2b endpoint — off-policy forward-KL KD through 32k context; the starting checkpoint for all stage-3 arms. Strong teacher-forced fit (wikitext-2 PPL 12.24 vs teacher 12.09; 0-shot avg 0.632) but broken generation…
Downloads · 30 days
20
23% of all-time downloads
All-time downloads
87
Public
Parameters
4.5B
9.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.1 GB · 100%
From the Hugging Face model README
Stage-2b endpoint — off-policy forward-KL KD through 32k context; the starting checkpoint for all stage-3 arms. Strong teacher-forced fit (wikitext-2 PPL 12.24 vs teacher 12.09; 0-shot avg 0.632) but broken generation behavior: MATH-500 thinking 0.000 (100% non-termination), RULER vt@32k 0.00 — the evaluation-blind-spot exhibit.
Part of a study converting full-attention Qwen3-4B into a GDN (gated
DeltaNet) hybrid (27 of 36 layers converted, uniform 1:4 retention) and
recovering capability via staged distillation, culminating in on-policy
distillation (OPD): student rollouts scored per-token by the frozen teacher
under reverse KL. Checkpoints are stock Qwen3NextForCausalLM — load with
AutoModelForCausalLM (transformers >= 4.57). vLLM serving needs a small
registry shim (all-dense MoE + tied-embeddings fixes) from the project repo.
Pipeline: surgery/init -> stage-1 hidden-state alignment (100M tok) -> stage-2a forward-KL KD @4k (600M) -> stage-2b long-context KD @32k (294M) -> stage-3 OPD (200M gen tokens, horizon 512->16k, token-keyed LR). Teacher/reference: Qwen/Qwen3-4B (thinking mode).