Qwen3.8-27B-absolute-heresy-W4A16

Qwen3.8-27B W4A16 (AutoRound)

A 4-bit weight-only quant of MuXodious/Qwen3.8-27B-absolute-heresy, packed in the compressed-tensors format and tuned for single-GPU inference on a 24 GB card (RTX 3090).

• Parameters: ~28 B (27.99 B logical; base model is 27B-class)
• On disk: 15.79 GB (14.71 GiB)
• Weights in VRAM: ~14.71 GiB — leaving room for the KV cache, activations, and CUDA graphs on a 24 GB card

Quantized with the script in syv-ai/qwen38-27b-rtx3090 → single-user.

Quantization

• Scheme: W4A16 — 4-bit integer weights, 16-bit activations
• Method: AutoRound, symmetric, group_size=128, pack-quantized
• Format: compressed-tensors (quant_method: compressed-tensors)
• Vision tower: left in bf16 (visual blocks are in the quant ignore list), so image understanding is preserved

Serving with vLLM

This is a multimodal (text + image) model. It also ships MTP draft weights, so you can run multi-token prediction for speculative decoding.

Quick start

vllm serve Starw1/Qwen3.8-27B-absolute-heresy-W4A16 \
  --gpu-memory-utilization 0.97 \
  --max-model-len 140000 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --limit-mm-per-prompt '{"image":4,"video":0}'

Links

• Source model: https://huggingface.co/MuXodious/Qwen3.8-27B-absolute-heresy
• Quantization script: https://github.com/syv-ai/qwen38-27b-rtx3090/tree/main/single-user

Quantized weights only — see the source model card for architecture, training, and license details.