Downloads · 30 days
361
6% of all-time downloads
mad-lab-ai/Qwen3.6-35B-A3B-tq-gguf
Qwen3.6-35B-A3B-tq-gguf is a machine learning model from mad-lab-ai. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Same-day TurboQuant (TQ) quantizations of Qwen/Qwen3.6-35B-A3B by mad-lab-ai.
Downloads · 30 days
361
6% of all-time downloads
All-time downloads
6.1K
Public
Repo size
56.8 GB
Likes
5
Public
Click a slice to open those files.
.gguf56.8 GB · 100%
From the Hugging Face model README
Same-day TurboQuant (TQ) quantizations of Qwen/Qwen3.6-35B-A3B by mad-lab-ai.
All quants use imatrix calibration generated from Python coding datasets for improved accuracy on code generation tasks.
| Quant | Size | bpw | Notes |
|---|---|---|---|
| TQ1_0 | 7.4 GB | 1.69 | Ternary — minimum footprint |
| TQ2_0 | 8.9 GB | 2.06 | Ternary — solid size/quality balance |
| TQ3_1S | 17 GB | 4.00 | WHT-rotated — recommended daily driver |
| TQ4_1S | 21 GB | 5.00 | WHT-rotated — highest quality |
TurboQuant applies a Walsh-Hadamard Transform (WHT) rotation to weights before quantization, resulting in better perplexity than standard GGUF formats at equivalent bit widths. TQ3_1S targets Q4_K_M quality at a meaningfully smaller footprint.
Qwen3.6-35B-A3B is a Mixture-of-Experts model with 35B total parameters but only ~3B active per token. This has an important practical implication for local inference:
When you offload layers to CPU RAM, only the active experts for each token are actually computed — the inactive expert weights sit in RAM but are never touched for that token. This means CPU offloading on a MoE model carries a much smaller performance penalty than on a dense model of equivalent size.
In practice: TQ3_1S at 17GB with a few layers offloaded to CPU on a 16GB GPU should still achieve 20+ t/s token generation — something that would be impractical with a dense 17GB model.
All quants use imatrix calibration data generated from Python coding datasets:
ajibawa-2023/Python-Code-23k-ShareGPTiamtarun/python_code_instructions_18k_alpacaflytech/python-codes-25k~5MB of Python code and instruction content, optimized for code generation workloads.
TurboQuant quants require a llama.cpp build with TQ support. Use one of these:
Standard llama.cpp builds will not recognize these quant types.
./build/bin/llama-server \
-m Qwen3.6-35B-A3B-TQ3_1S.gguf \
-ngl 99 \
-fa on \
-c 8192 \
--port 8080
./build/bin/llama-server \
-m Qwen3.6-35B-A3B-TQ3_1S.gguf \
-ngl 52 \
-fa on \
-c 8192 \
--port 8080
Adjust -ngl to fit your VRAM. Thanks to MoE architecture, even significant CPU offload maintains good throughput.
./build/bin/llama-server \
-m Qwen3.6-35B-A3B-TQ3_1S.gguf \
-ngl 99 \
-fa on \
-c 8192 \
-ctk tq3_0 -ctv tq3_0 \
--port 8080
| Quant | VRAM | Expected t/s | Notes |
|---|---|---|---|
| TQ1_0 | 8 GB | 25-35 t/s | Fits most consumer GPUs |
| TQ2_0 | 10 GB | 25-35 t/s | Good balance for 8-12GB cards |
| TQ3_1S | 16-18 GB | 20-30 t/s | 16GB GPU with minor CPU offload viable thanks to MoE |
| TQ4_1S | 24 GB | 18-25 t/s | Full fit on 24GB GPU |
Estimates based on similar MoE models. Your results will vary by hardware.
Quantized by mad-lab-ai