Gordon-35B-A3B-mixed-q2k-imat

λ Gordon — Qwen3.6-35B-A3B Mixed q2_K + imatrix

"CPU-offload-aware" GGUF · the fast generalist of the Black Mesa mixed-quant series.

A custom mixed-precision GGUF of Qwen3.6-35B-A3B, hand-tuned for a single 18 GB dual-GPU desktop. It delivers 256K context at ~78 tok/s decode — roughly the speed you'd otherwise only get at 64K — by quantizing only the expert layers that get offloaded to system RAM down to Q2_K, while keeping everything that lives on the GPU at Q4_K. Measured perplexity is within 0.6% of the lossless Q8_0.

TL;DR: a 17 GB file that runs the full 256K window at small-context speed, because the bytes that the CPU has to stream each token were cut roughly in half — and an importance matrix keeps the quality cost negligible.

---

Why this exists

On a RAM-bandwidth-bound machine (the experts that don't fit in VRAM are streamed from DDR4 every token), decode speed is dominated by how many bytes per token the CPU reads, not by the total file size. A uniform IQ4_XS of this model runs ~51 tok/s at 256K because ~14 expert layers are offloaded at 4.25 bpw. If you make those specific offloaded layers Q2_K (~3.0 bpw, roughly half the bytes) and leave the GPU-resident tensors at Q4_K, the per-token RAM traffic drops sharply and decode jumps to ~78 tok/s — the same throughput as the 64K profile, but at the full 256K context.

The trick is alignment: the low-bit layers must be exactly the layers you pin to CPU. This build quantizes blocks 13–26 (the ffn_(up|gate|down)_exps tensors) to Q2_K and you launch with a matching -ot override that pins those same tensors to CPU. The total file size is almost irrelevant; only the offloaded-layer byte count matters.

Recipe

• Base: Qwen3.6-35B-A3B (hybrid qwen35moe: gated attention + gated-delta-net SSM layers, 256 experts, 8+1 active, ~3B active params/token).
• Source: the lossless Q8_0 GGUF, requantized with llama-quantize.
• Mixed layout:
  • ffn_*_exps on blocks 13–26 → Q2_K (42 tensors — the CPU-offloaded set)
  • all other weights → Q4_K (334 tensors, GPU-resident)
  • output / embedding-class tensors → Q6_K (56 tensors)
• Importance matrix: computed on a held-out calibration corpus (~61K tokens / 119 chunks) and used to guide the Q2_K rounding. The imatrix lowers perplexity 0.58% vs a blind Q2_K build at identical speed (2.4195 vs 2.4337).
• Result: 4.88 bpw effective, ~17 GB on disk, 40 blocks, 256K native context.

Benchmarks

Measured on RTX 3060 Ti (8 GB) + RTX 3080 (10 GB) = 18 GB VRAM, Ryzen 5950X, 46 GB DDR4-2733, with ik_llama.cpp (fused-MoE, IQK kernels), q4_0 KV cache, flash-attention on.

Perplexity A/B vs the Q8_0 reference: +0.59% (both within the ±0.06 confidence interval — effectively lossless for a 2.4× decode speedup and 4× the context).

How to run (ik_llama.cpp)

The -ot override below is required — it pins the Q2_K layers (13–26) to CPU so the low-bit tensors are exactly the offloaded ones. Do not add --n-cpu-moe with this build.

./llama-server \
  -m Qwen3.6-35B-A3B-mixed-q2k-imat.gguf \
  --jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
  --ctx-size 262144 --parallel 1 --n-gpu-layers 99 \
  -ot 'blk\.(1[3-9]|2[0-6])\.ffn_(up|gate|down)_exps\.weight=CPU' \
  --tensor-split 44,56 --ubatch-size 256 --no-mmap --threads 8 --no-warmup \
  --port 8000

• --tensor-split 44,56 balances the two cards (8 GB / 10 GB). Adjust for your GPUs.
• On a single 24 GB GPU you don't need the offload at all — drop the -ot line and the whole model fits in VRAM (this quant is then just a small, fast Q4-class model).
• Works with mainline llama.cpp too, but ik_llama.cpp is ~2.8× faster at prefill on this hybrid MoE architecture.

Intended use & limitations

• Target: local inference on ~16–18 GB VRAM where you want the full 256K window without dropping to single-digit-context speeds. General assistant / coding / long-context use.
• The Q2_K expert layers are the quality floor; on heavy reasoning the difference vs Q8_0 is measurable but small (≈0.6% PPL). If you have the VRAM/RAM headroom and want maximum fidelity, use a Q8_0 or Q6_K build instead.
• Inherits all capabilities and biases of the base Qwen3.6-35B-A3B. No additional fine-tuning or alignment changes were made — this is purely a quantization.

Provenance

• Base model: Qwen3.6-35B-A3B by Qwen (Apache-2.0).
• Quantization, imatrix, mixed layout, and tuning: xero0000, June 2026.
• Tooling: llama-quantize / llama-imatrix (llama.cpp family) for the build; ik_llama.cpp for serving.

Quantization does not change the model's license; this derivative is released under the base model's Apache-2.0 license.

---

Open long-context recipe (hybrid GDN, measured 2026-08-07)

This quant is a hybrid qwen35moe model: most sequence mixing is gated-delta-net / SSM (fixed-size recurrent state) with full softmax attention only on an interval of layers (full_attention_interval). That is the same class of physics as long-context hybrids (not 10M dense full-KV).

What is open

• Weights: this mixed GGUF (Apache-2.0 base: Qwen3.6-35B-A3B)
• Runtime: ik_llama.cpp / fork with GDN + FA paths (expert-streaming-engine fa-q4-d256-longctx)
• Serving recipe: 3× consumer GPUs (~26 GiB), q4_0 KV, flash-attn on

Measured on this hardware (2026-08-07)

Hardware: RTX 3060 Ti 8 GB + RTX 2080 Super 8 GB + RTX 3080 10 GB, ik_llama.cpp, slot qwen36-mixed-3gpu / one-shot 512K.

Train n_ctx = 262144. Serving at 524288 allocates a larger virtual window; quality beyond train is extrapolated (RoPE / hybrid state) and is not a RULER claim.

Exact launch flags (3-GPU all-VRAM 256K production)

CUDA_VISIBLE_DEVICES=<3060ti>,<2080s>,<3080> \
llama-server -m Qwen3.6-35B-A3B-mixed-q2k-imat.gguf \
  --jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
  --ctx-size 262144 --parallel 1 --n-gpu-layers 99 --ctx-checkpoints 8 \
  --split-mode layer --tensor-split 31,31,38 \
  --ubatch-size 512 --no-mmap --threads 8 --no-warmup --port 8000

Higher virtual window (512K) one-shot

# fewer checkpoints to save VRAM; n_ctx above train
llama-server ... --ctx-size 524288 --ctx-checkpoints 2 --ubatch-size 256 ...

vs closed “10M on one 4090” claims

Closed systems that advertise multi‑million context on a single 24 GB GPU do not store 10M tokens of full softmax KV. They use fixed-state / hybrid sequence mixers. This open recipe is the same physics class, with honest measured ceilings on ~26 GB multi-GPU hardware.