Downloads · 30 days
9
27% of all-time downloads
arianraje/qwen3-4b-gdn-otmix-2p0b-opd
qwen3-4b-gdn-otmix-2p0b-opd is a machine learning model from arianraje. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Model weights of the Qwen3-4B GDN-hybrid on-policy-distillation (OPD) WSD ladder run qwen-wsd-otmix-h32k-2p0b-v1 (W&B id qwen-otmix-h32k-b200-2p0b-v1-20260904) at step 11701 / 2,000,169,520 consumed tokens, H=32K.
Downloads · 30 days
9
27% of all-time downloads
All-time downloads
33
Public
Parameters
4.5B
9.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.1 GB · 100%
From the Hugging Face model README
Model weights of the Qwen3-4B GDN-hybrid on-policy-distillation (OPD) WSD ladder run qwen-wsd-otmix-h32k-2p0b-v1
(W&B id qwen-otmix-h32k-b200-2p0b-v1-20260904) at step 11701 / 2,000,169,520 consumed tokens, H=32K.
Ladder: OpenThoughts prompt mix (stage3_prompts_v1, ~15% RUG / ~20% general), horizon 32,768, 4x B200 (3-GPU ZeRO-1 trainer + 1 vLLM sampler, gen_batch 256). Each rung = ~200M flat-LR tokens at 2e-5 plus an 839-step (110M-token) linear decay, seeded from the previous rung's pre-decay end-minus-0839 full state; the first rung seeds from wsd-flat-ext800-predecay-890M (pinkskin/qwen3-4b-gdn-wsd-ladder).
LR tail: decayed to 0. Compare rungs with matching tails.
Decayed final of the wsd_qwen_otmix_h32k_b200x4_2p0b rung; the pre-decay full state (end-minus-0839) is kept locally and not uploaded.
Custom GDN-hybrid architecture - register before loading (see project repo).
Source Git commit: d86fbef09d35f4e4d7943ec51d2b3732eb1fed46