Downloads · 30 days
15
10% of all-time downloads
omshunyaom13/qwen3.6-27b-autoround-int4-mm-fable
qwen3.6-27b-autoround-int4-mm-fable is a text generation model from omshunyaom13. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Qwen3.6-27B Fable finetune — AutoRound INT4 quantized, with MTP speculative decoding.
Downloads · 30 days
15
10% of all-time downloads
All-time downloads
150
Public
Parameters
5.9B
18.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.1 GB · 100%
How the weights are stored.
I323.1B · 53%
From the Hugging Face model README
Qwen3.6-27B Fable finetune — AutoRound INT4 quantized, with MTP speculative decoding.
This is the production-ready INT4 checkpoint of the Fable finetune of Qwen3.6-27B, repackaged into the nested multimodal-wrapper shape so vLLM's built-in MTP (Multi-Token Prediction) speculative decoding fires at 100% acceptance. Served text-only via --language-model-only (the vision tower is a meta-device placeholder — zero VRAM).
| Metric | Value |
|---|---|
| Base model | Qwen3.6-27B (Fable finetune) |
| Quantization | AutoRound INT4 (weights), fp8_e5m2 KV cache |
| Speculative decoding | MTP, accept-len 4.00 @ 100% (3 draft positions) |
| Decode throughput | ~57–73 tokens/sec (TP=2) |
| Prefill throughput | ~1650 tok/s @ 10K tokens, ~1347 tok/s @ 90K |
| Time-to-first-token | ~172–185 ms |
| VRAM / card (voice profile) | ~20.2–20.7 GB |
| VRAM / card (base profile) | ~23.8–24.0 GB |
| Power / card | ~331 W @ 100% util |
| Soak test | 0 MiB growth, 0 errors, 100% TPS retention |
| Quality (benchlocal --medium) | 66/75 (88%) — ToolCall 93%, InstructFollow 93% |
vLLM v0.24.0, TP=2, dual GPU:
vllm serve /path/to/model \
--tensor-parallel-size 2 \
--language-model-only \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--quantization auto-round \
--max-model-len 262144
Critical: keep mtp.fc in BF16. Quantizing the MTP head (mtp.fc) drops acceptance to 0%. The INT4 quantization must exclude mtp.fc (keep it in BF16) — the rest of the MTP layers can stay INT4.
The model uses the multimodal-wrapper architecture (Qwen3_5ForConditionalGeneration) with text_config.mtp_num_hidden_layers=1. Served text-only (--language-model-only) — the vision tower is a meta-device (no VRAM, no weights).
Dual-GPU without NVLink: set NCCL_P2P_DISABLE=1 (PCIe path).
Measured on dual NVIDIA GeForce RTX 3090 PCIe (no NVLink), 24 GB/card, 330 W power cap.
Two serving configurations (mutually exclusive on a single GPU pair):
| Profile | Util | Context | VRAM/card | Purpose |
|---|---|---|---|---|
| voice | 0.78 | 131K | ~20.2 GB | Co-resident TTS + STT (frees ~3.9 GB/card) |
| base | 0.92 | 262K | ~23.8 GB | Full context, max throughput |
Dropping utilization from 0.90→0.78 costs zero decode TPS — decode is weight/MTP-bound, not KV-bound.
Apache-2.0 (base model: Qwen3.6-27B Apache-2.0; Fable finetune: Apache-2.0).