Qwen3.8-Flash-Next-DS4-Q4

Qwen3.8-Flash-Next DS4 Q4 (imatrix)

DS4 quantization of Qwen/Qwen3.8-Flash-Next, built from the official BF16 checkpoint at revision de4b8e4d43b917e7706784d8bb445c9af86a3540 with an importance-matrix calibrated Q4_K routed-expert recipe.

The current release is the qwen4exp-schema structure: standard llama.cpp GGUF tensor naming that the DS4 engine (Metal graph) loads directly, with the 51B-value PLE n-gram table kept in the external sidecar so it never has to be memory-resident.

The main GGUF includes the 48 trunk layers and one embedded MTP block. This is the only model version distributed here: use the same file for ordinary decoding or enable speculative decoding with --mtp. The external PLE table is required in both cases.

Quantization recipe

The importance matrix is the pinned unsloth/Qwen3.8-Flash-Next-GGUF calibration (revision c8b5954a88c2775c54692593eda40ea041d3176, imatrix SHA-256 a5863123db1ca458727e738955befb7bfc199520aa2bee3a30142a1aff925415d, dataset unsloth_calibration_Qwen3.8-Flash-Next.txt, 45 chunks of 18432). 24 zero-count part-experts (16 gate/up, 8 down) use the deterministic per-expert input-column weight-energy fallback; all other entries are normalized per expert.

Note: the Hub sidebar's auto-detected quantization label keys off the PLE sidecar (genuinely Q4_1); the routed experts are Q4_K/MXFP4 as listed above.

Files

Running

Requires the DS4 runtime from ivanfioravanti/ds4, branch qwen3.8-flash-next (make, run from the repo root).

# Download the main model and the required PLE table.
QWEN_DS4_DIR="$HOME/models/qwen38-ds4-q4"
hf download ivanfioravanti/Qwen3.8-Flash-Next-DS4-Q4 \
  --local-dir "$QWEN_DS4_DIR"

# Speculative decoding using the MTP block inside the main GGUF.
./ds4 \
  -m "$QWEN_DS4_DIR/Qwen3.8-Flash-Next-Q4KImatrixExperts-MXFP4Down-BF16Emb-BF16Control-Q8GDN-Q8QSA-Q8Shared-Q8Out-MTP.gguf" \
  --ple "$QWEN_DS4_DIR/Qwen3.8-Flash-Next-PLE-Q4_1.gguf" \
  --mtp --metal --power 100 --mtp-draft 7 --mtp-timing

# Ordinary decoding uses the same GGUF, with MTP disabled.
./ds4 \
  -m "$QWEN_DS4_DIR/Qwen3.8-Flash-Next-Q4KImatrixExperts-MXFP4Down-BF16Emb-BF16Control-Q8GDN-Q8QSA-Q8Shared-Q8Out-MTP.gguf" \
  --ple "$QWEN_DS4_DIR/Qwen3.8-Flash-Next-PLE-Q4_1.gguf" \
  --metal --power 100

Pass the MTP-inclusive GGUF as -m / --model and enable its embedded block with --mtp. Do not pass this file through --mtp-model: that external-support option is not supported for Qwen3.8.

--mtp-timing reports draft acceptance. The current Qwen graph uses one draft token per verification cycle; --mtp-draft 7 is accepted but does not enable seven-token draft chains. Add --temp 0 for greedy decoding, or --mtp-exact-sampling to preserve the target distribution when sampling.

--ple is required because the main model carries no inline n-gram table. The sidecar mapping is CPU-only and never enters Metal residency. The conversion tool that produced these files from the fast-pack artifacts is gguf-tools/qwen4_pack_to_qwen4exp.py in the same repository (--ple-external, --mtp).

Requirements

• Apple Silicon with Metal (developed and measured on an M3 Ultra, 512 GB).
• Resident model weights occupy approximately 69.7 GiB, including MTP, plus the context-dependent KV cache and graph buffers. Use a 128 GB or larger Mac for headroom.
• The PLE sidecar stays on disk and is demand-paged (32.0 GB / 29.8 GiB, fast local SSD recommended).

Measured on the reference M3 Ultra

Short-context structured prompts (e.g. math) reach ~66 tok/s with MTP at ~98% draft acceptance; the numbers above are prose-continuation workloads. MTP preserves exact sampling with --mtp-exact-sampling.

Quality

Scored against an exact-checkpoint local BF16 reference (100-case fixture, 24-token greedy continuations, top-20 logprobs; 85 tokenization-aligned cases): target-logprob MAE 0.0431, greedy first-token match 74/85, top-1 agreement 96.6%, average target NLL 0.213. For reference on the same harness, the ggml-org Q8_0 build scores MAE 0.0211 / top-1 98.7%, and the previous Q4_0-routed release scored MAE 0.0430 / 96.5% — the imatrix Q4_K recipe buys back the down-projection precision budget as MXFP4 while holding overall drift flat. Methodology lives in the repository's QWEN38_FLASH_NEXT.md and OUT/qwen38-eval/ reports.

Attribution and license

Derived from Qwen/Qwen3.8-Flash-Next (thanks to the Qwen team), with the imatrix calibration from unsloth/Qwen3.8-Flash-Next-GGUF and the DS4 inference engine by the ds4 contributors (engine: Tarjei Mandt). This derivative inherits the Qwen Community License 1.0 of the base model — the full text is in LICENSE and the usual license conditions (attribution, branding, and the MaaS / AI Work Assistant clauses) apply as they do to the original.