Downloads · 30 days
19
2% of all-time downloads
soyrsoyr/Qwen3.6-27B-W4A16-AWQ-GPTQ
Qwen3.6-27B-W4A16-AWQ-GPTQ is a text generation model from soyrsoyr. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Quantized version of Qwen/Qwen3.6-27B using AWQ + GPTQ W4A16 quantization.
Downloads · 30 days
19
2% of all-time downloads
All-time downloads
770
Public
Parameters
27B
43.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors25.9 GB · 100%
How the weights are stored.
I3218.8B · 69%
From the Hugging Face model README
Quantized version of Qwen/Qwen3.6-27B using AWQ + GPTQ W4A16 quantization.
| Benchmark | Score (flexible) | Score (strict) |
|---|---|---|
| GSM8K | 70.05% | 68.54% |
Evaluated using lm_eval with the HuggingFace (transformers) backend.
This model was quantized using llm-compressor v0.12.0 with a two-stage pipeline:
duo_scaling="both" with a 20-point grid search per layer.block_size=128, dampening_frac=0.01, and actorder="static".AWQModifier:
duo_scaling: both
n_grid: 20
GPTQModifier:
targets: [Linear]
ignore: [lm_head, "re:.*linear_attn.*"]
scheme: W4A16
block_size: 128
dampening_frac: 0.01
actorder: static
Qwen3.6 is a vision-language model, but this checkpoint contains only the quantized text sub-model. Use --language-model-only to serve it with vLLM:
vllm serve soyrsoyr/Qwen3.6-27B-W4A16-AWQ-GPTQ \
--language-model-only \
--max-model-len 4096 \
--max-num-seqs 256 \
--gpu-memory-utilization 0.9
Tested on vLLM 0.25.1 with a single H100 80GB.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"soyrsoyr/Qwen3.6-27B-W4A16-AWQ-GPTQ",
device_map="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("soyrsoyr/Qwen3.6-27B-W4A16-AWQ-GPTQ")
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(output[0], skip_special_tokens=True))
lm_head and all linear_attn layers are excluded from quantizationrope_parameters field (containing mrope_section) from the original VL config has been removed since it is not needed for text-only inference and causes an M-RoPE assertion error in vLLMTwo bugs in transformers prevent loading this model without patches:
generation/configuration_utils.py: model_config.to_dict() called on a raw dict (line ~1247)quantizers/quantizer_compressed_tensors.py: .base_model_tp_plan accessed on a dict from get_text_config()Both stem from qwen3_5_text model type returning a dict instead of a config object from get_text_config().
soyrsoyr - Red Hat