Qwen3-Coder-480B-A35B-Instruct-GGUF

This is an IQ1_M_R4 quant of Qwen3-Coder-480B-A35B-Instruct, using the imatrix from ubergarm, with a slightly modified recipe:

# Attention
blk\..*\.attn_q.*=iq6_k
blk\..*\.attn_k.*=q8_0
blk\..*\.attn_v.*=q8_0
blk\..*\.attn_output.*=iq6_k

# Routed Experts
blk\..*\.ffn_down_exps\.weight=iq1_m_r4
blk\..*\.ffn_(gate|up)_exps\.weight=iq1_m_r4

# Non-Repeating Layers
token_embd\.weight=iq4_k
output\.weight=iq6_k

The file size is 112749523904 bytes, or 105.006 GiB.

With the routed experts fully offloaded to CPU, the memory usage (before context) will be:

llm_load_tensors:        CPU buffer size = 97863.13 MiB
llm_load_tensors:  CUDA_Host buffer size =   500.77 MiB
llm_load_tensors:      CUDA0 buffer size =  9156.73 MiB

I have a code refactoring eval that uses greedy decoding with deterministic output, and this is the smallest quant I tested that can still perform the same as the BF16 version (openrouter, alibaba provider).

Note that this quant only works with ik_llama.cpp.

Big thanks to ubergarm for sharing the imatrix, the recipes, and most importantly the detailed guidance and thought processes.