Downloads · 30 days
40
18% of all-time downloads
altomek/Qwen3-30B-A3B-XXL-GGUF
Qwen3-30B-A3B-XXL-GGUF is a text generation model from altomek. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This is Q6K GGUF quant with some extra params. I just thought - can I make such a cool quant Unsloth is doing? I took a look at tensors in their GGUF models and noticed differences in standard quants also took a look…
Downloads · 30 days
40
18% of all-time downloads
All-time downloads
217
Public
Repo size
29.5 GB
Likes
0
Public
Click a slice to open those files.
.gguf29.5 GB · 100%
From the Hugging Face model README
This is Q6_K GGUF quant with some extra params. I just thought - can I make such a cool quant Unsloth is doing? I took a look at tensors in their GGUF models and noticed differences in standard quants also took a look at llama-quantize options and noticed some cool options I can use to produce something simmilar. Made few qauants and vibe tested them on some tricky prompts to find configuration that pass them all successfuly while both ordinary Q6_K and UD-Q6_XL quants do not pass them. BTW, GGUF is very flexible format and this is what I also learned with this experiment. I thought only Exllama have flexibility but it is not true and with GGUF you also can optimise quants for your need. What is special here? All attention related layers are bf16, few first and last layers are Q8_0 and also all ffn_down_exps are Q8_0... so quite a lot Q8_0 layers. I would say it is Q7_K quant ;P Here is llama-quantize command used:
llama-quantize --leave-output-tensor --output-tensor-type q8_0 --token-embedding-type q8_0 --tensor-type attn_q=bf16 --tensor-type attn_v=bf16 --tensor-type attn_k=bf16 --tensor-type attn_output=bf16 --tensor-type blk.0.ffn_gate_exps=q8_0 --tensor-type blk.0.ffn_up_exps=q8_0 --tensor-type blk.1.ffn_gate_exps=q8_0 --tensor-type blk.1.ffn_up_exps=q8_0 --tensor-type blk.2.ffn_gate_exps=q8_0 --tensor-type blk.2.ffn_up_exps=q8_0 --tensor-type blk.27.ffn_gate_exps=q8_0 --tensor-type blk.27.ffn_up_exps=q8_0 --tensor-type blk.28.ffn_gate_exps=q8_0 --tensor-type blk.28.ffn_up_exps=q8_0 --tensor-type blk.29.ffn_gate_exps=q8_0 --tensor-type blk.29.ffn_up_exps=q8_0 --tensor-type ffn_down_exps=q8_0 ..\model.gguf ..\..\models\Qwen3-30B-A3B-Q6_K_XXL.gguf Q6_K 12"
XXL GGUF quant is a joke. Nothing new here, I just used params llama-quantize already have. However I find out MoE models and especialy this one are highly sensitive to quantization. Gol was to have something smaller then Q8_0 but still performing exeptionally like if it was unquantized model... I think this quant is realy, realy good. Do you share my vibe tested view? :P Have fun with this model!