Qwen3.6-27B-PrismaQuant-5.1bit-Hybrid

Qwen3.6-27B-PrismaQuant-5.1bit-Hybrid

A 5.1 bits-per-weight, mixed-precision ("Hybrid") quantization of Qwen3.6-27B, built with PrismaQuant and served on vLLM (compressed-tensors). Precision is allocated per layer by measured sensitivity (Fisher diagonal + validated output-KL): NVFP4 (W4A4) on robust layers, FP8 on sensitive ones, BF16 on the most sensitive (embeddings, norms, lm_head, MTP head). Average 5.1 bpp. Per-layer map is in mixed_native_manifest.json.

Why it exists

At the time of upload, it scores the highest of any Qwen 3.6 27B in the best agentic tool-calling bench.

Evaluation

Measured on tool-eval-bench 2.0.7, deterministic harness (prefix-cache off, temp 0, single-stream, reference-validated in-session), identical serve recipe for both models:

Headline = full tool-eval across categories; hardmode = the hard split. Numbers are on one pinned harness version and are not comparable to scores from other versions.

Serving (vLLM)

vllm serve <path-or-repo> \
  --dtype bfloat16 --gpu-memory-utilization 0.85 --max-model-len 32768 \
  --trust-remote-code --attention-backend flashinfer \
  --enable-chunked-prefill --no-enable-prefix-caching \
  --quantization compressed-tensors --kv-cache-dtype fp8 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Notes

• Speculative decoding: the model ships with an MTP head. Use num_speculative_tokens=2 — k=2 maximizes decode throughput on tested hardware; k=3 is net overhead at the current acceptance rate. (The MTP head may be updated with a version trained against the quantized weights for higher acceptance.)
• Hardware: built and validated on a DGX Spark (GB10, sm_121a). It's a standard compressed-tensors checkpoint and should serve on other vLLM-supported GPUs.
• Quantization format: compressed-tensors, mixed FP8 + NVFP4.
• License: inherits the base model (Qwen/Qwen3.6-27B) — confirm/set the correct license tag before public release.