Qwen3.6-35B-A3B-Q8_0-dequant-bf16-vllm

Qwen3.6-35B-A3B Q8_0 GGUF Dequantized BF16 for vLLM

This private artifact was converted from unsloth/Qwen3.6-35B-A3B-GGUF file Qwen3.6-35B-A3B-Q8_0.gguf into standard Hugging Face safetensors for vLLM serving.

The converter streams GGUF tensors, dequantizes them to fp32, applies the Qwen3.6/GDN tensor remaps, performs the required RMSNorm offset handling, fuses MoE gate and up expert tensors into mlp.experts.gate_up_proj, then casts once to BF16 before writing sharded safetensors.

Validation

• GGUF dry-run mapping: 733 input tensors, 0 skipped, 0 unmapped.
• Output index: 693 HF keys, 21 safetensor shards, 69,321,221,376 bytes.
• Transformers load: passed as AutoModelForImageTextToText with expected missing visual tower weights only.
• Numeric spot-checks: passed with max_abs_vs_bf16=0 for sampled direct, norm-offset, reshape/GDN, and fused expert tensors.
• Transformers generation smoke: passed.
• vLLM smoke: passed on vLLM 0.23.0 with --language-model-only.

vLLM command used

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/model   --served-model-name qwen36-35b-a3b-gguf-q8_0-bf16   --dtype bfloat16   --max-model-len 4096   --gpu-memory-utilization 0.85   --attention-backend triton_attn   --gdn-prefill-backend triton   --moe-backend triton   --trust-remote-code   --language-model-only

The visual tower files are intentionally absent because the GGUF source contains language weights only; use vLLM text serving with --language-model-only.