Downloads · 30 days
1.2K
1% of all-time downloads
soyrsoyr/Qwen3.8-27B-W4A16-AWQ-GPTQ
Qwen3.8-27B-W4A16-AWQ-GPTQ is a image-text-to-text model from soyrsoyr. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
Quantized version of Qwen/Qwen3.8-27B, a Qwen3.5-family hybrid-attention VLM, produced with llm-compressor and served with vLLM.
Downloads · 30 days
1.2K
1% of all-time downloads
All-time downloads
94.5K
Public
Parameters
11.1B
27.7 GB on disk
Likes
9
Public
Click a slice to open those files.
.safetensors27.7 GB · 100%
How the weights are stored.
I3218.8B · 68%
From the Hugging Face model README
Quantized version of Qwen/Qwen3.8-27B, a Qwen3.5-family hybrid-attention VLM, produced with llm-compressor and served with vLLM.
Linear layers only. The vision tower
(re:.*visual.*), the hybrid linear-attention mixers (re:.*linear_attn.*),
and lm_head stay in bf16. The full VLM (with vision_config) is saved in the
compressed-tensors format,
and the base MTP predictor is preserved for speculative decoding.from transformers import AutoModelForImageTextToText, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
MODEL_ID = "Qwen/Qwen3.8-27B"
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
# ... load 512 ultrachat_200k samples, chat-templated, at 2048 tokens ...
recipe = [
AWQModifier(duo_scaling="both", n_grid=20),
GPTQModifier(targets=["Linear"], scheme="W4A16", block_size=128,
dampening_frac=0.01, actorder="static",
ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn.*"]),
]
oneshot(model=model, dataset=ds, recipe=recipe,
max_seq_length=2048, num_calibration_samples=512)
model.save_pretrained("Qwen3.8-27B-W4A16-AWQ-GPTQ", save_compressed=True, save_original_format=False)
tokenizer.save_pretrained("Qwen3.8-27B-W4A16-AWQ-GPTQ")
vllm serve soyrsoyr/Qwen3.8-27B-W4A16-AWQ-GPTQ
from vllm import LLM, SamplingParams
llm = LLM(model="soyrsoyr/Qwen3.8-27B-W4A16-AWQ-GPTQ")
out = llm.generate(["The capital of France is"], SamplingParams(max_tokens=32))
print(out[0].outputs[0].text)
Recovery vs. the bf16 base, evaluated through the vLLM backend with lm-evaluation-harness (OpenLLM v1) and lighteval (generative reasoning at temperature 0.6, top_p 0.95, up to 32k tokens).
| Benchmark | Qwen3.8-27B | W4A16-AWQ-GPTQ | Recovery |
|---|---|---|---|
| ARC-Challenge (25-shot), acc_norm | 50.68 | 50.09 | 98.8% |
| HellaSwag (10-shot), acc_norm | 71.99 | 71.94 | 99.9% |
| TruthfulQA-mc2 (0-shot), acc | 61.25 | 60.33 | 98.5% |
| Winogrande (5-shot), acc | 76.87 | 76.64 | 99.7% |
| Average | 65.20 | 64.75 | 99.3% |
MMLU and GSM8K are omitted. Qwen3.8-27B is a reasoning model, so under the OpenLLM v1 protocol GSM8K has its chain of thought truncated and MMLU's loglikelihood is measured where the model wants to emit its think block, both of which collapse to a harness artifact rather than a real score. Math and knowledge are captured by the generative reasoning suite instead.
| Benchmark | Qwen3.8-27B | W4A16-AWQ-GPTQ | Recovery |
|---|---|---|---|
| AIME-24, avg@4 | 95.00 | 90.83 | 95.6% |
| AIME-25, avg@4 | 93.33 | 85.00 | 91.1% |
| MATH-500, pass@1 | 82.00 | 80.80 | 98.5% |
| Average | 90.11 | 85.54 | 94.9% |
AIME-24 and AIME-25 are reported as avg@4 (mean accuracy over 4 samples per problem, temperature 0.6, up to 32k tokens, stderr about 4 points) to average out the large single-sample variance of a 30-problem test. The base and the quantized model are scored in the same run under identical settings, so the small remaining gap reflects the int4 quantization rather than sampling noise. MATH-500 (500 problems) is pass@1.
User-reported serving results, not verified by me.
4x RTX 3090 24GB, vLLM, TP4 (from discussion #1, thanks to @mwyborski):