Qwen3.8-Queen-27B-W4A16-AWQ

Qwen3.8-Queen-27B-W4A16-AWQ

W4A16 (4-bit weights, 16-bit activations) AWQ compressed-tensors quantization of aifeifei798/Qwen3.8-Queen-27B.

Quantization method

• Scheme: W4A16_ASYM — 4-bit asymmetric per-group quantization (group size 128) of all Linear weights, stored in the compressed-tensors pack-quantized format (weight_packed / weight_scale / weight_zero_point / weight_shape), which LMDeploy turbomind auto-detects and loads natively (including the MTP heads and vision tower, which stay BF16).
• Tooling: llmcompressor one-shot offline quantization with CPU offloading (compressed_tensors.offload.load_offloaded_model), so the full-precision source fits on a 2×16 GB VRAM setup.
• AWQ activation smoothing: AWQModifier with the layer-scoped hybrid-attention mappings from build_hybrid_attention_mappings — full-attention input_layernorm → self_attn.q/k/v, post_attention_layernorm → mlp.gate/up, and mlp.up_proj → mlp.down_proj, with duo_scaling="both" and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.
• Unquantized (kept BF16): embeddings, lm_head, norms, linear_attn.in_proj_a/b, the vision tower, and MTP heads.
• Calibration: 128 chat samples from HuggingFaceH4/ultrachat_200k (default train_sft split) — chosen over STEM-skewed instruction data to match the model's open-domain creative/roleplay focus.

Usage

Tested with LMDeploy turbomind:

from lmdeploy import pipeline, TurbomindEngineConfig

pipe = pipeline(
    "TheUnderscore/Qwen3.8-Queen-27B-W4A16-AWQ",
    backend_config=TurbomindEngineConfig(
        tp=2,
        model_format="compressed-tensors",
        language_model_only=True,
        cpu_realtime_conversion=True,
    ),
)
print(pipe("Hello, who are you?").text)

Files

• quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards). --offload_dir selects where per-rank CPU offload temp folders live (defaults to the current working directory).
• model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16 so the full model architecture is loadable.