Downloads · 30 days
9
29% of all-time downloads
arianraje/qwen3-4b-gdn-hybrid-stage3-200M-OPD-dtfix
qwen3-4b-gdn-hybrid-stage3-200M-OPD-dtfix is a machine learning model from arianraje. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Model weights of the Qwen3-4B GDN-hybrid dtbias-fixed line, stage-3 on-policy distillation (OPD), 200M generated tokens.
Downloads · 30 days
9
29% of all-time downloads
All-time downloads
31
Public
Parameters
4.5B
9.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.1 GB · 100%
From the Hugging Face model README
Model weights of the Qwen3-4B GDN-hybrid dt_bias-fixed line, stage-3 on-policy distillation (OPD), 200M generated tokens.
Run stage3-opd-dtfix-200M-v1 (W&B id qwen-wsd-flat-dtfix-200M-v1-20260903), final snapshot at step 1247 / 200,104,080 consumed generated tokens, horizon 16K, flat LR 2e-5 with a 150-step warmup and a 300-step (39.3M-token) linear decay to 1e-6.
Student: arianraje/qwen3-4b-gdn-hybrid-stage2b-kd-dtfix (stage-2b KD on the dt_bias-fixed stage-1 alignment arianraje/qwen3-4b-gdn-hybrid-stage1-align-dtfix). Teacher: Qwen/Qwen3-4B. This is a like-for-like rerun of the ORIGINAL stage3-200M-OPD rung of pinkskin/qwen3-4b-gdn-wsd-ladder (wsd-flat-200M, WSD_exps/configs/wsd_flat.yaml): same prompt mixture (stage3_prompts_v1), validation set, schedule, gen_batch and queue factor. The only intended difference in the whole line is the GDN dt_bias initialisation: the original line inherited the transformers dt_bias.fill_(1.0) init (saturated decay gates); this line uses the fla-style init (dt_bias median -4.56).
The result JSONs are included (likelihood_battery.json, reasoning_bench_*.json); the head-to-head against the original line is in artifacts/dual_family_eval_24h/TABLES_DTFIX.md of the project repo.
Custom GDN-hybrid architecture - register before loading (see project repo).
Source Git commit: d86fbef09d35f4e4d7943ec51d2b3732eb1fed46