Qwen3.6-27B-MTP-48Gb

Qwen3.6-27B-MTP-48Gb

Overview

This repository hosts a production-ready Q8_0 quantization of Qwen3.6-27B, specifically optimized to run within 48 GB of VRAM (dual RTX 3090s via NVLink). The weights were calibrated using a custom iMatrix blending code, English text, agentic prompts, and French passages to precisely target layer-wise quantization sensitivity. On the tested hardware, this configuration achieves up to ~2,000 t/s prefill and ~85–90 t/s decode throughput with Multi-Token Prediction (MTP) enabled. The Q8_0 format was selected for its optimal speed-to-accuracy ratio, lower-precision variants (e.g., Q6_K) showed a 5 to 10% throughput drop alongside reduced long-context stability, capping reliable usage at ~242k tokens. By preserving critical precision in attention and KV cache operations, this quantization enables stable inference at the model’s maximum context window.

Note: Due to VRAM constraints, the MMPROJ vision encoder must be offloaded to CPU when running multimodal tasks.

Research & Methodology

Selective precision Quantization for high-speed inference

As previously said, Q8_0 was the way to go but the model became too big for the 48Gb target. Therefore, the strategy was to use Q6_K on just a few tensors. I managed to identify which were the least sensitive to quantization thanks to the iMatrix, which also identified the few tensors that needed more care in order to maintain the model's capacities. Therefore, the schema identified was :

Therefore, 93.91% of the model is Q8_0, 5.44% is Q6_K and 0.61% is F16.

iMatrix Calibration

The model was calibrated using a custom, shuffled iMatrix to ensure high fidelity across coding, instruction-following, and bilingual tasks (English/French).

Accuracy Analysis

To evaluate the precision loss after the mixed-quantization scheme, I compared the perplexity on Wiki-Text-raw and a custom made dataset, composed of code mainly found on llama.cpp's repo:

Key Findings:

• Near-Lossless: The perplexity degradation is minimal at +0.000709 on Wiki-Text-Raw, indicating that this mixed precision layout preserves the original model's reasoning and coding capabilities.
• Reaches full-Q8_0 capacities: Other tests made on a 'pure' Q8_0 quantization shown the same perplexity and KL-Div results, proof that this work has achieved its goals.

Recommended Usage

To replicate the optimal performance (262K context, F16 K-Cache, Multi-GPU) using llama.cpp, use the following llama-server command. Note the specific use of --split-mode tensor and --tensor-split 1,1 for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink. As previously disclosed, the mmproj file will not hold on the GPU, if you want to use it you need to add --no-mmproj-offload and --image-min-tokens 1024 to the following command, alongside the --mmproj /path/to/mmproj-BF16.gguf.

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/Qwen3.6-27B0-MTP-Q8-Opt.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --spec-type draft-mtp \
    --alias Qwen3.6-27b \
    --jinja

I also developed a proxy to enable users to select thinking or non-thinking behaviors, apply the recommended sampling parameters AND the "Preserve Thinking" option. You may find it on my GitHub.

Hardware Requirements

• Target VRAM: 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
• RAM: Minimum 32GB system RAM (Prompt caching, system overhead and MMPROJ).
• Context limit: The command above loads ~21GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing --ctx-size or using 8-bit K-cache (--cache-type-k q8_0).

Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

• The Qwen Team: For researching and releasing the exceptional Qwen3.6 architecture.
• llama.cpp: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
• Unsloth: For the immense work done on fixing Qwen3.5 and 3.6 chat_template and the base BF16 model used for this project.