Qwen3.8-Flash-Next-131B-A6B-GGUF

Qwen3.8-Flash-Next-131B-A6B (n-gram table pruned, GGUF)

A 64.8 GB GGUF of Qwen/Qwen3.8-Flash-Next that scores at parity with the full 90 GB quant on tool-calling, GSM8K, and MMLU — 28% smaller, with zero training and zero transformer surgery. A 75.4 GB Q4_0 variant of the same cut is also available for older GPUs; see Files.

Qwen3.8-Flash-Next carries a 51B-parameter n-gram hash-embedding table (~29% of total weights): 16 independent hash heads, 8 per n-gram type (bigrams / trigrams), injected at layer 1. This model keeps 1 head of 8 per n-gram type and drops the other 14, exploiting the built-in redundancy of multi-head hashing (each n-gram is looked up several independent ways; a Bloom-filter-style safety margin). Every other tensor is copied byte-for-byte from unsloth's UD-Q3_K_XL dynamic quant, so unsloth's imatrix calibration is preserved exactly.

Files

Same surgery, two different base quants. Pick by hardware, not by size.

The Q4_0 build is bigger, not smaller. The n-gram table is what makes this model compact; Q4_0 simply stores every other tensor at more bits than Q3_K_XL does. You are buying throughput with VRAM. On gfx906 that trade is usually worth it — Q4_0 reaches llama.cpp's MMVQ path while K-quants and IQ4_NL fall through to a generic dequant path several times slower.

The Q4_0 build has not been separately benchmarked, and no one has yet reported running it. The surgery is byte-verbatim and structurally verified (dropped heads point at a shared all-zero row that decodes to exact zeros in Q4_0, kept heads are byte-identical to the source), but the numbers below come from the Q3_K_XL build and do not transfer. If you run it, please report back in the discussions.

Benchmarks

All rows measured on the same llama.cpp build (b10673, CUDA, A100), temp 0, same day. Tool-calling: 40 agentic cases scored on calling the right tool with the right arguments or correctly declining (schemas disjoint from anything the surgery could see — there is no training step). GSM8K-15 / MMLU-30 subsets, no-think mode. Perplexity: wikitext-2 test, 32×512-token chunks.

Tool accuracy has a ±3-case run-to-run band across hardware backends at temp 0 (a CPU control run scored the base at .975) — read "1.000 vs .900" as parity with base, not as beating it. The one real cost is perplexity: the n-gram table turns out to be a surface-level next-token predictor. Removing 87.5% of it doubles wikitext perplexity while leaving reasoning and tool use at baseline.

For calibration: both cheaper cuts were also tested and rejected. Removing transformer layers (the classic prune) collapsed this architecture without healing — GSM8K fell to 0.33–0.67 and perplexity hit 13+ at 53–60 GB. The n-gram cut is the only free lunch here.

Running it

Needs llama.cpp from 2026-08-27 or newer (qwen4exp support; b10673 tested). Vendor-recommended sampling: instruct temp 0.7, top_p 0.8, presence_penalty 1.5; thinking temp 1.0, top_p 0.95.

# fully offloaded (~66 GB+ VRAM for Q3_K_XL, ~77 GB+ for Q4_0)
llama-server -m qwen38-keep1-Q3KXL.gguf -ngl 99 -c 8192

# smaller GPUs: keep the (sparse-gather) n-gram table in system RAM
llama-server -m qwen38-keep1-Q3KXL.gguf -ngl 99 -c 8192 \
  --override-tensor "per_layer_token_embd\.weight=CPU"

# CPU-only boxes work too (slow but correct), given free RAM to spare
# beyond the file size — ~70 GB for Q3_K_XL, ~80 GB for Q4_0

The --override-tensor flag matters more on the Q4_0 build: it moves the table (3.6 GB) to system RAM and leaves ~72 GB of transformer on the GPU, which is what makes 2×32 GB cards viable at all.

Text-only GGUF (vision tensors ship separately as unsloth's mmproj; the vision path is untested with this surgery). MTP speculative decoding is not available — upstream llama.cpp does not export or run the MTP head for this architecture yet.

Limitations

• Wikitext perplexity 4.66 vs 2.40 for the base: prose is measurably less "polished-autocomplete" even though task performance holds. If your workload is verbatim-recall-heavy (quotes, boilerplate reproduction), the missing table may show up.
• Evaluated on a small in-house benchmark (40 tool cases, GSM8K-15, MMLU-30) at ≤8K context. Long-context behavior (QSA sparse attention, 256K native) untested.
• The kept heads were chosen positionally (first of each 8), not by importance ranking. A calibrated head choice might do slightly better; nobody has measured.
• The Q4_0 build is unbenchmarked and, as of upload, unrun by anyone. It is structurally verified but not behaviourally verified — treat it as a build to test rather than a build to rely on until someone reports back.

How this model was made

Full write-up, surgery script, and raw result logs: github.com/Cyronius/model-experiments (surgery_qwen38.py).

Short version: the GGUF's per-head hash-table layout is entirely metadata-driven (ple.head_offsets, ple.head_vocab_sizes), and llama.cpp reads the table's row count back from the tensor itself. Dropped heads get vocab_size = 1 pointing at a shared all-zero row, so they contribute nothing and cost one row of storage — no llama.cpp patch, no requantization, no training. The whole build is a streaming byte copy: ~45 minutes on an NVMe laptop, and the entire experimental campaign that selected this configuration cost $1.50 of A100 time.

The all-zero row works because it decodes to exact zeros in every quant type used here: an IQ4_NL block with zero bytes, and a Q4_0 block whose d scale is zero, both yield 0.0 for all 32 elements. That is what makes the same script reusable across base quants — it reads the table's type out of the source file and byte-slices accordingly, so producing the Q4_0 variant needed no code change at all.

License

qwen-community-1.0, inherited from Qwen/Qwen3.8-Flash-Next. Quantized weights derived from unsloth/Qwen3.8-Flash-Next-GGUF (Q3_K_XL build) and bartowski/Qwen3.8-Flash-Next-GGUF (Q4_0 build) — thanks to both for the calibrated quants these builds inherit byte-for-byte, imatrix work included.