Downloads · 30 days
193
7% of all-time downloads
hugypufy/Qwen3.6-35B-A3B-MXFP4_MOE-GGUF
Qwen3.6-35B-A3B-MXFP4_MOE-GGUF is a image-text-to-text model from hugypufy. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is an optimized GGUF of noctrex/Qwen3.6-35B-A3B-MXFP4MOE-GGUF Q80 variant.
Downloads · 30 days
193
7% of all-time downloads
All-time downloads
2.7K
Public
Repo size
41.1 GB
Likes
3
Public
Click a slice to open those files.
.gguf41.1 GB · 100%
From the Hugging Face model README
This is an optimized GGUF of noctrex/Qwen3.6-35B-A3B-MXFP4_MOE-GGUF Q8_0 variant.
The only change from the source model is the re-quantization of the output.weight (lm_head) tensor from BF16 to a more compact format. Two variants are provided:
During token generation (batch size 1), this matrix is read in full for every single token, making it a significant bandwidth bottleneck. The smaller variants reduce memory bandwidth pressure proportionally, yielding higher token generation throughput with no measurable impact on prompt processing speed.
Benchmarks done on llama.cpp with two AMD R9700 GPUs
| Metric | Original (BF16 output) | Q8_0 output | Change |
|---|---|---|---|
| TG128 | 104.0 t/s | 113.6 t/s | +9.2% |
| TG2048 | 103.8 t/s | 113.5 t/s | +9.3% |
| TG8192 | 102.0 t/s | 111.0 t/s | +8.8% |
| PP128 | 1507 t/s | 1527 t/s | +1.3% |
| PP2048 | 3878 t/s | 3906 t/s | +0.7% |
| PP8192 | 5379 t/s | 5361 t/s | -0.3% |
| Metric | Original (BF16 output) | MXFP4 output | Change |
|---|---|---|---|
| TG128 | 104.0 t/s | 119.3 t/s | +14.7% |
| TG2048 | 103.8 t/s | 119.1 t/s | +14.7% |
| TG8192 | 102.0 t/s | 116.6 t/s | +14.3% |
| PP128 | 1507 t/s | 1538 t/s | +2.1% |
| PP2048 | 3878 t/s | 3903 t/s | +0.6% |
| PP8192 | 5379 t/s | 5383 t/s | +0.1% |
| Metric | BF16 output (18.87 GiB) | Q8_0 output (18.42 GiB) | MXFP4 output (18.17 GiB) |
|---|---|---|---|
| TG128 | 104.0 t/s | 113.6 t/s | 119.3 t/s |
| TG2048 | 103.8 t/s | 113.5 t/s | 119.1 t/s |
| TG8192 | 102.0 t/s | 111.0 t/s | 116.6 t/s |
| PP128 | 1507 t/s | 1527 t/s | 1538 t/s |
| PP2048 | 3878 t/s | 3906 t/s | 3903 t/s |
| PP8192 | 5379 t/s | 5361 t/s | 5383 t/s |
output.weight which is re-quantized (Q8_0 or MXFP4)The MoE expert weights in this model use MXFP4 (4-bit) quantization, which is very compact. However, the final language modeling head (output.weight) was left in BF16 — a 248320×2048 matrix consuming ~970 MiB. During token generation (batch size 1), this matrix is read in full for every single token, making it a significant bandwidth bottleneck.
Profiling with llama.cpp's PERF_LOGGER showed that the output.weight matrix-vector multiply consumed ~16% of total token generation time, entirely bottlenecked by memory bandwidth.
By re-quantizing this single tensor to a more compact format, the bandwidth savings translate directly into faster token generation.
Q8_0 (block size 32, int8 + f16 scale per block) shrinks the tensor to ~515 MiB — a ~47% reduction. This is a high-fidelity quantization format (1 byte per weight + a per-block f16 scale). For a projection matrix of this size (508M elements), the quantization noise introduced by Q8_0 is negligible. Informal text generation tests show no perceptible quality degradation.
MXFP4 (shared FP8 scale per 32-element block, 4-bit elements) shrinks the tensor to ~257 MiB — a ~73% reduction. This provides the smallest possible output tensor size, yielding the highest token generation throughput. The MXFP4 format matches the quantization already used for the MoE expert weights in this model. While this is a more aggressive quantization than Q8_0, informal text generation tests suggest only subtle quality differences in practice.
Acknowledgments Special thanks to noctrex for the source MXFP4 MoE GGUF quantization that this model is derived from.