qwen36-eagle3-drafter-v5

zenith1232/qwen36-eagle3-drafter-v5

EAGLE-3 speculative-decoding draft head for Qwen/Qwen3.6-35B-A3B, trained with a Mixture-of-Experts draft block (not the usual dense draft).

Architecture

• 1 decoder block, hidden_size=2048, head_dim=256
• Full-attention self-attn (num_heads=16, num_kv_heads=2)
• MoE MLP:
  • num_experts=32, top_k=4
  • moe_expert_intermediate=2048
  • shared_expert: True
  • aux-free load balance (DeepSeek-V3 style) + router z-loss (ST-MoE)
• aux_hidden from target layers 3 / 19 / 35
• draft vocab: 32000 (shortlist of most frequent 32K target tokens)

Training

• Teacher: Qwen/Qwen3.6-35B-A3B (bf16)
• Data: ~126M tokens of teacher trajectories (Stage B, 437750 records)
• Loss: progressive TTT (length 7, ploss weights 0.8^k) + MoE router z-loss
• Best ckpt val_ce0: 1.4727 at step 4000

Serving

Not yet supported in vanilla vLLM — the upstream Eagle3 loader assumes a dense draft MLP. A ~100 LOC patch is needed to route through the MoE block (in progress). Until then, load the raw safetensors via transformers and run inference through the Eagle3MoE module in our training repo.

Files

• model.safetensors — trained weights, _orig_mod. prefix stripped
• config.json — Eagle3MoE config (hidden_size, num_experts, top_k, etc)
• best_info.json — step + val_loss + val_ce0 of the saved checkpoint