Downloads · 30 days
23
1% of all-time downloads
Qwen3.6-27B-INT8-Autoround-V2
Minachist/Qwen3.6-27B-INT8-Autoround-V2
Qwen3.6-27B-INT8-Autoround-V2 is a image-text-to-text model from Minachist. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
This is an unofficial mixed-precision INT8 quantized version of Qwen3.6-27B. It was created using AutoRound.
All-time downloads
2.5K
Public
Parameters
9.1B
30.9 GB on disk
Likes
7
Public
Repo makeup
Click a slice to open those files.
.safetensors30.9 GB · 100%
Parameter types
How the weights are stored.
I326.2B · 67%
Try a prompt
Base models
Model card
From the Hugging Face model README
Qwen3.6-27B-INT8-Autoround-V2
This is an unofficial mixed-precision INT8 quantized version of Qwen3.6-27B. It was created using AutoRound.
Quantization details
Some layers are not quantized as quantizing them causes a heavy downgrade, or AutoRound simply doesn't support quantizing them.
| Precision | Layer Count | Target Layers |
|---|---|---|
| INT8 (group_size=-1, symmetric) | 410 | gate/up/down_proj, q/k/out_proj, in_proj_qkv, in_proj_z, mtp |
| FP16 (unquantized) | 98 | in_proj_a / in_proj_b of linear_attn, embed_tokens, visual |
- AutoRound doesn't support quantizing
in_proj_aandin_proj_b. - Calibration configuration:
iters=250,nsamples=1024,batch_size=4, withgradient_accumulate_steps=2.
You need to change a line in utils.py of the AutoRound library
from:
layer_config[match] = val
to:
layer_config[match] = copy.deepcopy(val)
in order to quantize linear_attn.
To whoever wants to open a pull request upstream: if you believe this is actually a bug, I encourage you to do so.
Evaluation Results
1. KL Divergence (KLD)
The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.
| Metric @ctx 512 | Value |
|---|---|
| Mean KLD | 0.00182 ± 0.00014 |
| Median KLD | 0.000672 |
| P90 / P95 / P99 KLD | 0.00232 / 0.00349 / 0.00931 |
| P99.9 / Max KLD | 0.0668 / 9.156 |
| Top-1 Agreement | 98.32% |
| PPL (Base → Quantized) | 7.7415 → 7.7442 |
| ln(PPL Ratio) | 0.000343 |
Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.
- I heard quantizing linear_attn impacts the performance quite badly, but that's not the case here. Maybe not quantizing
in_proj_a/in_proj_bis the reason.
2. Benchmarks
Measured via lm-evaluation-harness (0.4.12) using the vLLM OpenAI-compatible API.
- Settings: 0-shot,
temperature=0.6, top_p=0.95, max generation tokens 65,536. - Comparison baseline: Official metrics from the Qwen3.6-27B model card.
| Benchmark | This Model | Base Model (Official) | Difference | n |
|---|---|---|---|---|
| MMLU-Redux | 91.9% ± 1.6 | 93.5 | -1.6 | 285 |
| MMLU-Pro | 83.8% ± 1.8 | 86.2 | -2.4 | 420 |
- Performance drops in knowledge-focused benchmarks (MMLU-Redux, MMLU-Pro) are minor and lie mostly within the standard error. I wanted to run more benchmarks, but it's quite hot here and AC doesn't seem to work well. You have to do it yourself.
How to Use
This model has been tested using the latest vllm-openai:nightly.
On a 2x RTX 3090 setup, you can get 204,800 context without kv-cache quantization.
You can append --kv-cache-dtype fp8 to optimize memory further.
Acknowledgements
- Alibaba / Qwen team for the base Qwen3.6-27B model.
- Intel AutoRound team for the quantization framework.
- vLLM project for the inference engine and Qwen3.x MTP support.
- Task
- Image-Text-to-Text
- Library
- transformers
- Type
- qwen3_5
- License
- apache-2.0
- Created
- Jul 21, 2026
- Updated
- Jul 30, 2026