Qwen3.6-35B-A3B-NVFP4-MTP-ModelOpt

Qwen3.6-35B-A3B-NVFP4-MTP-ModelOpt

NVFP4 (nvfp4) quantization of Qwen/Qwen3.6-35B-A3B produced with NVIDIA TensorRT Model Optimizer via a multi-node ModelOpt quantizer. KV cache FP8. This is a quantized derivative and inherits the base model's license; see the base model card for usage.

<!-- BEGIN quant:preview -->

Quantization details (auto-generated)

• source model: Qwen/Qwen3.6-35B-A3B
• qformat: nvfp4 kv_cache: fp8
• calibration: ? samples from ?
• producer: NVIDIA ModelOpt ?
• generated: ?

Before/after sample generation was skipped for this run (SKIP_GENERATE=1).

<!-- END quant:preview --> <!-- BEGIN quant:remarks -->

Notes

Quantized MTP/NEXTN draft head (research probe)

Unlike a BF16-draft-head build, this checkpoint NVFP4-quantizes the MTP/NEXTN draft head (a full-attention + shared-expert-MoE block) and bakes FP8 KV scales onto its attention, so the SGLang p51 KV-scale loader exercises the MTP head. Because transformers drops the MTP layer at load and the multinode FSDP2 export carries no mtp.* at all, the head is produced by a post-export surgical requant that sources the raw BF16 mtp.* from the original checkpoint and NVFP4-packs them, mirroring each module's main-model analogue. All three gates have now been measured on GB10/sm121: the smoke-serve loads register-free with NEXTN active, GSM8K scores 96.5% (see the benchmarks table), and the draft head accepts 2.71 of a theoretical 3.00 tokens per step under load. What is still missing is the A/B against a BF16-MTP build, so the numbers below establish that the quantized draft works, not that it is free.

MTP scales are heuristic, not calibrated

The NVFP4 weights of the draft head are data-free and bit-identical to a calibrated run, but its W4A4 input_scale is a generous-amax heuristic and its FP8 KV k_scale/v_scale are borrowed from the last main-model full-attention layer. Correctness is unaffected (spec-decoding verifies every draft token); a poor draft only lowers acceptance/throughput. Measured: mean accept len 2.71 of a theoretical maximum of 3.00 (num_steps=2, num_draft_tokens=3), i.e. 90% of the ceiling, so whatever the heuristic scales cost is bounded by the remaining 10%. A BF16-MTP A/B would pin that number down exactly, but the headroom it could recover is small.

Multimodal, vision tower kept BF16

Qwen3.6-35B-A3B carries a 27-layer vision tower. Calibration is text-only, so the vision stack is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode); this is why the model must be quantized via the multinode driver, whose quant policy applies the vision excludes.

Hybrid-arch serving knobs

Serving needs the mamba scheduler knobs to avoid a spec-v2-vs-radix-cache boot crash: mamba_scheduler_strategy extra_buffer plus SGLANG_ENABLE_SPEC_V2=1, along with the flashinfer attention backend and flashinfer_cutlass MoE/FP4-GEMM backends. NEXTN speculative decoding at num_steps=2 was the validated throughput sweet spot; higher step counts (3/4/5) decayed.

Serving requires SGLang runtime patches (stock SGLang does not load this)

Stock SGLang 0.5.15 crashes loading this checkpoint on GB10/sm121; it needs five launch-time source patches, shipped in the public dgxarley repo (https://github.com/vroomfondel/dgxarley) under roles/k8s_dgx/files/sglang_patches/ and applied by sglang_launch.sh: (1) p50 -- allow QUANTIZED attention for modelopt_fp4 (this is a uniform-W4A4 export that also quantizes the Gated-DeltaNet linear-attn, unlike NVIDIA's MoE-only NVFP4); (2) p51 -- load the main model's baked FP8 KV scales onto RadixAttention (remap ...self_attn.k_proj.k_scale -> ...attn.k_scale), else the full-attn layers default to scale 1.0; (3) p43 -- keep quant_config for the QUANTIZED MTP draft head, else SGLang assumes a BF16 MTP, builds it unquantized, and the fused-MoE loader narrows the unpacked intermediate dim over the NVFP4-packed expert weight ("start(0)+length(512) exceeds dimension size(256)"); (4) p44 -- apply the same KV-scale mapper in the MTP load path (qwen3_5_mtp.py has its own load_weights) so the draft attention loads its baked FP8 scales instead of the 1.0 sentinel; (5) the arch-independent sm121 CUTLASS-FP4 mma patch. p43/p44 are specific to a quantized MTP and are inert for a BF16-MTP checkpoint. Independently, config.json pins torch_dtype: bfloat16 (the Qwen3.6 source config omits it) so SGLang's --dtype auto resolves bf16; without it the unquantized in_proj_ba loads as fp16 and crashes at the CUDA-graph capture ("mat1 and mat2 ... BFloat16 != Half"). Validated end to end on GB10/sm121 (image 0.5.15.post1-sm121): loads register-free, NEXTN speculation active, coherent output, MTP draft attn loads its borrowed KV scales (k_scale 0.0398, byte-identical to the last full-attn layer, the requant borrow source).

<!-- END quant:remarks --> <!-- BEGIN quant:benchmarks -->

Benchmarks

<!-- END quant:benchmarks -->