Qwen3.5-397B-A17B-GGUF

ik_llama.cpp imatrix Quantizations of Qwen/Qwen3.5-397B-A17B

NOTE ik_llama.cpp can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants. Only a couple quants in this collection are compatible with mainline llamma.cpp/LMStudio/KoboldCPP/etc as mentioned in the specific description, all others require ik_llama.cpp.

Some of ik's new quants are supported with Nexesenex/croco.cpp fork of KoboldCPP with Windows builds. Also check for ik_llama.cpp windows builds by Thireus here..

These quants provide best in class perplexity for the given memory footprint.

Big Thanks

Shout out to Wendell and the Level1Techs crew, the community Forums, YouTube Channel! BIG thanks for providing BIG hardware expertise and access to run these experiments and make these great quants available to the community!!!

Also thanks to all the folks in the quanting and inferencing community on BeaverAI Club Discord and on r/LocalLLaMA for tips and tricks helping each other run, test, and benchmark all the fun new models! Thanks to huggingface for hosting all these big quants!

Finally, I really appreciate the support from aifoundry.org so check out their open source RISC-V based solutions!

Quant Collection

Perplexity computed against wiki.test.raw. (lower is "better")

These two are just test quants for baseline perplexity comparison and not available for download here:

• BF16 738.493 GiB (16.005 BPW)
  • PPL over 580 chunks for n_ctx=512 = 3.4852 +/- 0.01883
• Q8_0 392.549 GiB (8.508 BPW)
  • PPL over 580 chunks for n_ctx=512 = 3.4862 +/- 0.01883

NOTE: The first split file is much smaller on purpose to only contain metadata, its fine!

IQ4_KSS 194.058 GiB (4.206 BPW)

PPL over 580 chunks for n_ctx=512 = 3.5102 +/- 0.01896

<details> <summary>👈 Secret Recipe</summary>

#!/usr/bin/env bash

custom="
# 60 Repeating Layers [0-59]

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq4_ks
blk\..*\.ffn_(gate|up)_exps\.weight=iq4_kss

# Non-Repeating Layers
token_embd\.weight=iq6_k
output\.weight=iq6_k
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

numactl -N ${SOCKET} -m ${SOCKET} \
./build/bin/llama-quantize \
    --custom-q "$custom" \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16.dat \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-IQ4_KSS.gguf \
    IQ4_KSS \
    128

</details>

Q3_K 179.97 GiB (3.90 BPW)

PPL over 580 chunks for n_ctx=512 = 3.5409 +/- 0.01924

This is a custom mainline llama.cpp compatible MoE optimized mix similar to AesSedai/ddh0's mixes and likely better than vanilla Q3_K_ mixes. Check the recipe for details.

<details> <summary>👈 Secret Recipe</summary>

#!/usr/bin/env bash

./build/bin/llama-quantize \
    --tensor-type ffn_down_exps=q4_K \
    --tensor-type ffn_gate_exps=q3_K \
    --tensor-type ffn_up_exps=q3_K \
    --token-embedding-type q4_K \
    --output-tensor-type q6_K \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16-mainline.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-Q3_K.gguf \
    Q8_0 \
    128

</details>

IQ2_KL 138.142 GiB (2.994 BPW)

PPL over 580 chunks for n_ctx=512 = 3.6536 +/- 0.02000

<details> <summary>👈 Secret Recipe</summary>

#!/usr/bin/env bash

custom="
# 60 Repeating Layers [0-59]

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq3_ks
blk\..*\.ffn_(gate|up)_exps\.weight=iq2_kl

# Non-Repeating Layers
token_embd\.weight=iq4_k
output\.weight=iq6_k
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

numactl -N ${SOCKET} -m ${SOCKET} \
./build/bin/llama-quantize \
    --custom-q "$custom" \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16.dat \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-IQ2_KL.gguf \
    IQ2_KL \
    128

</details>

smol-IQ2_XS 113.41 GiB (2.46 BPW)

PPL over 580 chunks for n_ctx=512 = 3.8717 +/- 0.02131

This is a custom mainline compatible MoE optimized mix similar to AesSedai/ddh0's mixes and likely better than vanilla mixes especially lacking imatrix. Check the recipe for details.

<details> <summary>👈 Secret Recipe</summary>

#!/usr/bin/env bash

./build/bin/llama-quantize \
    --tensor-type ffn_down_exps=iq2_xs \
    --tensor-type ffn_gate_exps=iq2_xs \
    --tensor-type ffn_up_exps=iq2_xs \
    --token-embedding-type q4_K \
    --output-tensor-type q6_K \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16-mainline.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-smol-IQ2_XS.gguf \
    Q8_0 \
    128

</details>

smol-IQ2_KS 108.142 GiB (2.344 BPW)

PPL over 580 chunks for n_ctx=512 = 3.9153 +/- 0.02172

<details> <summary>👈 Secret Recipe</summary>

#!/usr/bin/env bash

custom="
# 60 Repeating Layers [0-59]

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq2_ks
blk\..*\.ffn_(gate|up)_exps\.weight=iq2_ks

# Non-Repeating Layers
token_embd\.weight=iq4_k
output\.weight=iq6_k
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

numactl -N ${SOCKET} -m ${SOCKET} \
./build/bin/llama-quantize \
    --custom-q "$custom" \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16.dat \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-smol-IQ2_KS.gguf \
    IQ2_KS \
    128

</details>

smol-IQ1_KT 88.807 GiB (1.925 BPW)

PPL over 580 chunks for n_ctx=512 = 4.2523 +/- 0.02412

<details> <summary>👈 Secret Recipe</summary>

#!/usr/bin/env bash

custom="
# 60 Repeating Layers [0-59]

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq1_kt
blk\..*\.ffn_(gate|up)_exps\.weight=iq1_kt

# Non-Repeating Layers
token_embd\.weight=iq4_k
output\.weight=iq6_k
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

numactl -N ${SOCKET} -m ${SOCKET} \
./build/bin/llama-quantize \
    --custom-q "$custom" \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16.dat \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-smol-IQ1_KT.gguf \
    IQ1_KT \
    128

</details>

Quick Start

Example command for mainline llama.cpp for now including mmproj from another repo. Just remove the --mmproj if you don't want image capabilities.

# Clone and checkout
$ git clone https://github.com/ikawrakow/ik_llama.cpp
$ cd ik_llama.cpp

# Build for hybrid CPU+CUDA
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
$ cmake --build build --config Release -j $(nproc)

# Download Desired Quants
$ pip install huggingface_hub
$ hf download --local-dir ./ --include=smol-IQ2_XS/*.gguf ubergarm/Qwen3.5-397B-A17B-GGUF

# Hybrid CPU+GPU (ik_llama.cpp example for 96GB VRAM, adjust n-cpu-moe up for less VRAM)
model=/mnt/raid/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-Q3_K-00001-of-00005.gguf
./build/bin/llama-server \
    --model "$model"\
    --alias ubergarm/Qwen3.5-392B-A17B \
    --ctx-size 131072 \
    -sm graph \
    -smgs \
    -ngl 999 \
    -ts 48,48 \
    --n-cpu-moe 36 \
    -ub 4096 -b 4096 \
    --threads 24 \
    --host 127.0.0.1 \
    --port 8080 \
    --parallel 1 \
    --no-mmap \
    --jinja

# Hybrid CPU+GPU (mainline example here)
model=/mnt/raid/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-Q3_K-00001-of-00005.gguf
./build/bin/llama-server \
    --model "$model"\
    --mmproj /mnt/raid/models/ubergarm/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf \
    --alias ubergarm/Qwen3.5-392B-A17B \
    -fa on \
    --ctx-size 135168 \
    -ctk q8_0 -ctv q8_0 \
    -ub 2048 -b 2048 \
    -fit off \
    -ngl 999 \
    -ot "blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12)\.ffn_(gate|up|down)_exps.*=CUDA0,blk\.(47|47|48|49|50|51|52|53|54|55|56|57|58|59|60)\.ffn_(gate|up|down)_exps.*=CUDA1" \
    --cpu-moe \
    --threads 24 \
    --host 127.0.0.1 \
    --port 8080 \
    --parallel 1 \
    --no-mmap \
    --jinja

# CPU-Only (use ik_llama.cpp for fast chunked delta net implemention)
numactl -N "$SOCKET" -m "$SOCKET" \
./build/bin/llama-server \
    --model "$model"\
    --alias ubergarm/Qwen3.5-392B-A17B \
    --ctx-size 65536 \
    -ctk q8_0 -ctv q8_0 \
    -ub 4096 -b 4096 \
    --parallel 1 \
    --threads 96 \
    --threads-batch 128 \
    --numa numactl \
    --host 127.0.0.1 \
    --port 8080 \
    --no-mmap \
    --jinja

If you're using chat completions endpoint, you can disable thinking with --chat-template-kwargs '{"enable_thinking": false }'.

Vibe Coding

Seems like the autoparser branch is working fortunately!

• https://github.com/ggml-org/llama.cpp/pull/18675

You can get the freshest version like so:

git remote add pwilkin [email protected]:pwilkin/llama.cpp.git
git fetch pwilkin
git checkout pwilkin/autoparser

# compile as normal

Otherwise I noticed when trying opencode without autoparser branch it is spitting this error, still got the error using a chat template with --chat-template-file myTemplate.jinja from another repo.

Template supports tool calls but does not natively describe tools. The fallback behaviour used may produce bad results, inspect prompt w/ --verbose & consider overriding the template.
srv    operator(): got exception: {"error":{"code":500,"message":"\n------------\nWhile executing FilterExpression at line 120, column 73 in source:\n..._name, args_value in tool_call.arguments|items %}↵                        {{- '<...\n                                           ^\nError: Unknown (built-in) filter 'items' for type String","type":"server_error"}}

References

• ik_llama.cpp
• ubergarm on quantizing LLMs and tuning GPUs with aifoundry.org
• ubergarm-imatrix-calibration-corpus-v02.txt
• Getting Started Guide (out of date)
• Quant Cookers Guide (out of date)
• ik_llama.cpp Qwen3Next Issue
• ik_llama.cpp PR1288
• high quality imatrix MoE optimized mainline llama.cpp quants AesSedai/Qwen3.5-397B-A17B-GGUF
• high quality imatrix MoE optimized mainline llama.cpp quants tarruda/Qwen3.5-397B-A17B-GGUF
• amazing customized quants by Thireus (likely better than mine lol)