qwen38-flash-next-v100-recipe

Qwen3.8-Flash-Next — V100 recipe companion files

The two small files you need to run Qwen3.8-Flash-Next with MTP speculative decoding and vision on 2× Tesla V100-PCIE-32GB.

This repository deliberately does not host the model weights. It hosts only the two companion files that are easy to miss, plus a link to the full reproduction recipe.

Full recipe: github.com/mw00/peasant-smith — recipes/qwen38-flash-next-v100-32gb.md — exact llama.cpp build, CUDA/sm_70 setup, all flags, measured results.

What is here

What is NOT here (get the weights from upstream)

The 75.8 GB of quantized model weights live in the original repository. Download them from there:

hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
  --include "IQ3_XXS/*" --local-dir .

Source: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

That gives you two shards (00001-of-00002, 47.0 GB and 00002-of-00002, 28.8 GB). Note that shard 2 is not more weights — it is the 51.2B-parameter per-layer n-gram embedding table (per_layer_token_embd) at IQ4_NL. It is a lookup table, not matmul weights, so it is byte-identical across every quant variant.

The vision projector is also available upstream at the root of that repo. The copy here exists only so you can grab the two small files from one place.

Where the MTP head comes from: unsloth/Qwen3.8-Flash-Next-GGUF (MTP/). The shared-Q8_0 variant is the fastest of the set and is what is mirrored here.

Full download, all three sources

# 1. model weights (75.8 GB)
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
  --include "IQ3_XXS/*" --local-dir .

# 2. vision projector (0.91 GB)
hf download peasantsmith/qwen38-flash-next-v100-recipe \
  --include "GSQ-RCO-IQ3_XXS/*" --local-dir .

# 3. MTP draft head (2.60 GB)
hf download peasantsmith/qwen38-flash-next-v100-recipe \
  --include "MTP/*" --local-dir .

Requirements — read this before anything else

A stock ggml-org/llama.cpp build cannot use the MTP head. Mainline has no MTP graph for the qwen4exp architecture and no --spec-type draft-mtp option. Passing a head to it silently does nothing.

You need one of:

• The Unsloth prebuilt binaries — tag b10715-mix-86bd2d3 or newer, from unslothai/llama.cpp releases
• The upstream MTP pull request (ggml-org/llama.cpp#28243)

On a V100 (sm_70) specifically: pick the cuda12-portable bundle. The release ships several CUDA 12 assets and they do not all support Volta — the cuda12-newer bundle targets min sm 86 and will load then fail at device time. Check before running:

grep -E '^(variant|min sm|toolkit version):' BUILD_INFO.txt
# min sm must be <= 70

The recipe pins b11030-mix-5ff778e / cuda12-portable / commit 6ba30d0, which is what all the measurements were taken on.

Minimal working command

./llama-server \
  -m  IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
  --mmproj GSQ-RCO-IQ3_XXS/mmproj-Qwen3.8-Flash-Next-BF16.gguf \
  -md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 131072 -fa on \
  -ngl 99 -ncmoe 0 \
  --tensor-split 55,45 \
  -ctk bf16 -ctv bf16 -ctkd bf16 -ctvd bf16 \
  -t 12 -b 2048 -ub 512 \
  -lm mmap --lazy-mode on \
  --jinja --port 8080

Measured results (2× V100-PCIE-32GB, 64 GB total)

Pricing note (for anyone budgeting): used V100 prices fluctuate a lot and depend on seller, condition, and SXM2 vs PCIe. As of 23 September 2026, refurbished V100-PCIE-32GB cards were listed around £550 (~$735) each. Dated reference point only — check live listings.

The three things that cost the most time

1. min sm must be ≤ 70. Verify it in BUILD_INFO.txt, not from the version string. Wrong bundle = loads, then fails at device time.
2. -ncmoe 0 is mandatory. Without it experts spill to CPU and decode collapses from ~54 to ~19 tok/s.
3. The default tensor split is a trap. It loads with ~500 MB of headroom and OOMs once the KV cache grows. Use 55,45.

Everything else — CUDA 13 dropping Volta, the nvcc shadowing trap, KV quantization, n-max having opposite optima at short vs long context, and the full tuning matrix — is in the recipe.

Credit

• Model: ISTA-DASLab (GSQ/RCO non-uniform quantization)
• MTP draft heads: Unsloth
• Prebuilt MTP-capable llama.cpp: unslothai/llama.cpp
• Base model: Qwen