Qwen3.6-27B-W4A16-g32-asym

Qwen3.6-27B — GPTQ W4A16 Quantization (g32, Asymmetric)

This is a GPTQ W4A16 (4-bit weight-only) quantization of Qwen/Qwen3.6-27B, a 27B parameter hybrid linear-attention / full-attention multimodal model with vision, native tool calling, and chain-of-thought reasoning capabilities.

Quantization was performed using llm-compressor (v0.10.1.dev141) with the GPTQModifier, producing a compressed-tensors pack-quantized format that maps directly to vLLM's MarlinLinearKernel (fused INT4 GEMM) for maximum throughput. This quantization is specifically optimized for SDLC agent workloads (code generation, multi-turn tool calling) through task-specific calibration.

Model Details

Quantization Details

Recipe

default_stage:
  default_modifiers:
    GPTQModifier:
      config_groups:
        group_0:
          targets: [Linear]
          weights:
            num_bits: 4
            type: int
            symmetric: false          # Asymmetric: uses zero-points (int8)
            group_size: 32            # Fine-grained: 32-element scaling groups
            strategy: group
            actorder: static          # Static activation ordering
            observer: memoryless_minmax
      ignore:
        - lm_head
        - re:.*visual.*              # Full vision encoder preserved in BF16
        - re:.*linear_attn\.in_proj_a  # Linear attn projections (output_size=48)
        - re:.*linear_attn\.in_proj_b
      block_size: 128
      dampening_frac: 0.005          # Tight Hessian (5× default)
      actorder: static
      offload_hessians: false

Calibration Dataset

2,048 task-specific samples formatted with the Qwen3.6 ChatML template (<|im_start|>system\n...<|im_end|>). Task-specific calibration aligns the GPTQ Hessian estimation with the model's deployment distribution — particularly important for instruct models.

Quality Benchmarks

All benchmarks on wikitext-2-raw-v1 (test split), 639 non-overlapping chunks of 512 tokens, 296,434 tokens scored. Methodology consistent with llama.cpp ./perplexity -c 512 and the academic quantization literature. BF16 baseline PPL = 8.7116.

Comparative Quality

PPL = Perplexity (lower is better). Mean KLD = mean KL-divergence from BF16 output distribution (lower is better). Same-top-p = fraction of tokens where quantized model agrees with BF16 on the top predicted token (higher is better).

Throughput Benchmarks

Benchmarks measured on NVIDIA RTX 5090 (32 GB VRAM, sm120 / Blackwell) running vLLM with quantization: compressed-tensors, kv_cache_dtype: fp8_e4m3, max_model_len: 65536, single GPU (TP=1).

Single Request (concurrency=1)

20 requests, max_tokens=1024, temperature=0.9. Aggregate throughput is wall-clock total tokens / total wall time.

16 Concurrent Requests

160 total requests, max_tokens=1024, temperature=0.9.

Hardware Requirements

Minimum: 1× GPU with ≥32 GB VRAM (with reduced context window and fp8 KV cache).

The compressed-tensors pack-quantized format uses MarlinLinearKernel (fused INT4 GEMM) — requires NVIDIA GPU sm75+ (Turing and later). Works on: Ampere (A100, RTX 3090), Ada (RTX 4090), Hopper (H100, H200), Blackwell (RTX 5090, RTX PRO 6000).

Usage with vLLM

Tested with: vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 (vLLM v0.19.2rc1.dev134)

Docker Deployment (example with rope 384k context)

docker run -d --name vllm-qwen36-27b \
  --runtime=nvidia --gpus '"device=0"' \
  -p 8000:8000 \
  -v /path/to/model:/workspace/model \
  --ipc=host --shm-size=16g \
  --restart unless-stopped \
  vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 \
  /workspace/model \
  --host 0.0.0.0 --port 8000 \
  --quantization compressed-tensors \
  --dtype float16 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 393216 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 8 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image":4}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":1.5,"original_max_position_embeddings":262144}}}'

Example vLLM Configuration (YAML)

This configuration is deployed and verified on an RTX 5090 (32 GB):

# -- Model & Server ----------------------------------------------------------
model: /workspace/model
host: "0.0.0.0"
port: 8000
served_model_name: "qwen36-27b"
trust_remote_code: true
tensor_parallel_size: 1

# -- Quantization ------------------------------------------------------------
quantization: compressed-tensors

# -- Data Type ---------------------------------------------------------------
dtype: float16

# -- Context & Batching ------------------------------------------------------
max_model_len: 393216
max_num_batched_tokens: 8192
max_num_seqs: 8
enable_chunked_prefill: true

# -- Memory ------------------------------------------------------------------
gpu_memory_utilization: 0.95
enable_prefix_caching: true
kv_cache_dtype: fp8_e4m3

# -- Multi-Modal -------------------------------------------------------------
limit_mm_per_prompt:
  image: 4

# -- Tool Calling & Reasoning ------------------------------------------------
# qwen3_xml parser: handles <tool_call>...</tool_call> native format.
# Do NOT use qwen3_coder — it breaks on code containing angle brackets.
enable_auto_tool_choice: true
tool_call_parser: qwen3_xml
reasoning_parser: qwen3

# -- Thinking Mode -----------------------------------------------------------
# Disabled server-wide. Clients can override per-request:
#   extra_body={"chat_template_kwargs": {"enable_thinking": true}}
default_chat_template_kwargs:
  enable_thinking: false

# -- RoPE (REQUIRED for Qwen3.6 long context) --------------------------------
hf_overrides:
  text_config:
    rope_parameters:
      mrope_interleaved: true
      mrope_section: [11, 11, 10]
      rope_type: yarn
      rope_theta: 10000000
      partial_rotary_factor: 0.25
      factor: 1.5
      original_max_position_embeddings: 262144

# -- Misc --------------------------------------------------------------------
disable_custom_all_reduce: true
mm_encoder_tp_mode: data
mm_processor_cache_type: shm

Critical: The --hf-overrides / hf_overrides block with RoPE parameters is required for correct long-context behaviour. Without it, the model's YaRN scaling is not applied and context beyond ~256K tokens degrades.

Inference Test

# Text completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen36-27b","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

# Tool calling test
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-27b",
    "messages": [{"role":"user","content":"What is the weather in Prague?"}],
    "tools": [{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
    "tool_choice": "auto",
    "max_tokens": 256
  }'

Transformers / Python

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "gratex/Qwen3.6-27B-W4A16-g32-asym"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Explain GPTQ quantization in two sentences."}]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Files in This Repository

License

This quantization is released under the Apache 2.0 License, following the base model's license.

The base model Qwen/Qwen3.6-27B is licensed under Apache 2.0.

See LICENSE for the full license text.

Citation

If you use this model in your research or production systems, please cite:

@misc{qwen36-27b-gptq-w4a16-g32-asym,
  title  = {Qwen3.6-27B GPTQ W4A16 g32 Asymmetric Quantization},
  author = {Gratex International},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Qwen3.6-27B-W4A16-g32-asym}},
  note   = {Quantized with llm-compressor 0.10.1.dev141, task-specific calibration}
}

Acknowledgments

This quantization was produced using hardware and infrastructure provided by Gratex International, a.s.

---

Original Model: Qwen/Qwen3.6-27B
Quantization Tool: llm-compressor
Quantization Format: compressed-tensors
Deployment Engine: vLLM