Qwen3.6-35B-A3B-UD-Q4KM-dequant-bf16-vllm

Qwen3.6-35B-A3B UD-Q4_K_M GGUF dequantized to BF16 for vLLM

Source GGUF: unsloth/Qwen3.6-35B-A3B-GGUF, file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf.

Base config/tokenizer: Qwen/Qwen3.6-35B-A3B. Weights come from the GGUF, not the HF base checkpoint.

Conversion: GGUF tensors were dequantized to fp32, Qwen GDN layout transforms were reversed, Qwen RMSNorm offset tensors were converted to HF format, MoE gate/up expert tensors were fused into experts.gate_up_proj, and tensors were cast once to BF16 before writing sharded safetensors.

Validation on an H200 with vLLM 0.23.0:

• 733 GGUF tensors mapped to 693 safetensor keys because 40 expert gate/up tensor pairs are fused.
• Transformers CUDA load and short generation passed. The base visual tower is intentionally absent; serve this as text-only with --language-model-only.
• Numeric checks passed with max_abs_vs_bf16=0, including fused MoE gate/up halves; quantized matrix error vs fp32 direct dequant was BF16-scale (max_rel_vs_fp32 ~= 0.00389).
• vLLM /v1/completions smoke passed with Triton attention, Triton GDN prefill, and Triton MoE backend.

Recommended serve command:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve exolabs/Qwen3.6-35B-A3B-UD-Q4KM-dequant-bf16-vllm \
  --served-model-name qwen36-35b-a3b-gguf-udq4km-bf16 \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --attention-backend triton_attn \
  --gdn-prefill-backend triton \
  --moe-backend triton \
  --trust-remote-code \
  --language-model-only