Qwen3.5-397B-A17B-doublestuff

Qwen3.5-397B-A17B-doublestuff

Double-stuffed: a 198 GiB custom quant sized so that two full 256K-token q8_0-KV contexts fit alongside the weights in 256 GB of VRAM — verified on 8×32 GB AMD MI100.

A custom GGUF quantization of Qwen3.5-397B-A17B (MTP), built from unsloth/Qwen3.5-397B-A17B-MTP-GGUF BF16 shards using Unsloth's calibration imatrix. All credit for the BF16 conversion, the imatrix, and the calibration set goes to the Unsloth team — the only contribution here is where the bits go.

Why this exists

At 198 GiB it fills the gap in the standard lineup between UD-Q3_K_XL (170 GiB) and UD-Q4_K_S (217 GiB) — a size class Unsloth doesn't ship — chosen specifically for the two-concurrent-256K-session serving config in 256 GB. Every routed-expert tensor's bit-width is graded by imatrix activation importance (usage-weighted mean E[x²] per token, median-normalized across gate/up/down) rather than by layer position:

• the least-important layers go to q3_K,
• the highest-importance tail is promoted to q5_K,
• ffn_down (which feeds the residual stream) never drops below q4_K.

The imatrix says this model has no U-shape: importance spikes at blk.0–1, floors through blk.2–24, then rises monotonically ~350× toward blk.59. Grading by that measured curve — not the "middle layers matter least" folklore — is what earns the size class.

Perplexity (wikitext-2, 300 chunks @ 512, llama.cpp, one rig)

Honest guidance: if you have ≥217 GiB of VRAM, use Unsloth's UD-Q4_K_S — their dynamic allocation still wins at that budget. doublestuff is for max-context serving in 256 GB: it's the best PPL-per-byte in the set and beats the 15-GiB-larger flat-q4 build.

Why not just go smaller (vs UD-Q3_K_XL)?

UD-Q3_K_XL is 28 GiB smaller, but its entire routed-expert body is i-quant (IQ3_XXS on gate/up, IQ4_XS on down). doublestuff is pure K-quant, and on gfx908 (CDNA1) that wins on both axes:

Despite moving 14% fewer bytes/token, UD-Q3_K_XL decodes 12% slower — IQ codebook dequant is weakly optimized on CDNA1. On a GPU with fast IQ kernels the tradeoff differs; on MI100-class hardware, K-quant wins. Its smaller footprint buys neither accuracy nor speed.

Running it (8×MI100, gfx908, 256 GB VRAM)

Common flags: -ngl 999 --split-mode layer -fa on --no-mmap, HIP_VISIBLE_DEVICES=0,2,3,4,5,7,8,9 (adjust to your cards).

Only 15 of 61 blocks (every 4th, Gated-Attention) grow KV with context; the other 45 are Gated-DeltaNet with constant state — which is why 256K slots are this cheap.

Two serve modes

# A) single-stream, latency-optimized (chat / agentic)
-ngl 999 -sm layer -fa on --no-mmap -np 1 -ub 2048 \
  --spec-type draft-mtp --spec-draft-n-max 2

# B) max-context, two concurrent 256K sessions
-ngl 999 -sm layer -fa on --no-mmap -np 2 -c 524288 -ub 1024 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --tensor-split 0.98,1.13,1.13,1.12,1.07,0.92,0.82,0.95

• MTP self-speculation (--spec-type draft-mtp, the model's own blk.60 nextn head — no external draft model) speeds up single-stream decode, best at --spec-draft-n-max 2 (~+5% prose / +16% structured, lossless). n-max ≥ 3 loses to pipeline-verify cost, and plain n-gram speculation is a net loss on an all-GPU layer split. draft-mtp is -np 1 only, so it's mutually exclusive with mode B — pick latency or max-context.
• The tensor-split in mode B matters. The q5 tail + output + the MTP block make the last cards heavy, so a naive contiguous split OOMs at 2×256K. The split above (measured under a real long prefill, so the compute buffer is fully accounted) flattens the peak to 30.5 GiB @ ub512 / 31.6 GiB @ ub1024 of ~32. Single 256K contexts fit on an even split.
• Prefill micro-batch: the per-card compute buffer grows with -ub (~2.1 / 3.3 / 5.5 GiB at ub 512 / 1024 / 2048). In the double-slot config, -ub 2048 overflows the 256 GB aggregate — use -ub 1024 there (biggest prefill that fits both full slots) and reserve -ub 2048 for single-stream mode A, which has the headroom.

Recipe

Base ftype Q4_K_M; token_embd / output forced q6_K; routers kept F32; the MTP/nextn block (blk.60) at q4_K. Routed experts graded from the imatrix ranking:

• ffn_(gate|up)_exps: q3_K on the 38 least-important layers (blk1–38), q5_K on blk53–58, else q4_K.
• ffn_down_exps: q4_K floor everywhere, q5_K on blk50–59, q6_K on blk0.
• ffn_(gate|up)_shexp (always-active shared expert) and attn_output: q6_K.

Full recipe file (recipe_v2.txt) and the importance tooling (compute_importance.py, gen_recipe_v2.py) are in this repo.

Reproduction

Requires llama.cpp ≥ b3ce5ce (earlier llama-quantize rejects MTP models with Bad layer 60). Quantize from Unsloth's BF16 GGUF shards — their imatrix matches their conversion's layout; a fresh convert_hf_to_gguf.py run will not.

python3 compute_importance.py imatrix_unsloth.gguf   # -> importance.json
python3 gen_recipe_v2.py 38 6 10                      # -> recipe_v2.txt
llama-quantize \
  --imatrix imatrix_unsloth.gguf \
  --token-embedding-type q6_K --output-tensor-type q6_K \
  --tensor-type-file recipe_v2.txt \
  Qwen3.5-397B-A17B-BF16-00001-of-00018.gguf \
  Qwen3.5-397B-A17B-doublestuff.gguf Q4_K_M $(nproc)

Validation

Beyond the PPL table above, this quant was checked for: 2×256K double-slot serving (per-card VRAM measured under real long-prefill load), a K-quant-vs-IQ decode head-to-head vs UD-Q3_K_XL, MTP/n-gram/prefill-batch tuning sweeps, and an imatrix-weighted per-tensor quant-error study (which confirmed the allocation is near-optimal for layer-level grading). Details in the repo's analysis/ notes.

License

Inherits the base model's license (Qwen3.5, Apache-2.0). Credit to Qwen for the model and to Unsloth for the BF16 conversion, calibration set, and imatrix.