Qwen3.8-Flash-Next-MXFP4-FP8-R12

Qwen3.8-Flash-Next MXFP4–FP8 R12

This is an experimental mixed-precision conversion of Qwen/Qwen3.8-Flash-Next-FP8. It reduces the routed-expert checkpoint storage while retaining the 12 expert layers found most sensitive in our calibration and whole-model evaluation.

Run on one DGX Spark

GitHub recipe and benchmarks · Deployment details

Run on the DGX Spark:

git clone --branch qwen38-single-spark-release https://github.com/marco-jeffrey/spark-vllm-docker.git
cd spark-vllm-docker
./run-recipe.sh qwen3.8-flash-next-mxfp4-fp8-r12 --setup --solo

Setup builds the patched image on eugr's Spark base and reuses the standard Hugging Face cache. Allow approximately 146.55 GB for checkpoint files, plus Docker images and runtime caches. While this model is private, authenticate with hf auth login using an authorized account.

Quantization format

This is not a uniformly four-bit model:

• 36 of 48 main routed-expert layers use MXFP4 E2M1 weights with UE8M0 microscales;
• their activations are dynamically quantized to MXFP8 at inference time;
• 12 main routed-expert layers remain in the source block-FP8 format;
• the MTP routed experts and the 51.2B-value PLE n-gram table remain FP8; and
• attention, GDN, QSA, vision, embeddings, shared experts, routers, norms, hyperconnections, and the LM head retain their source checkpoint formats.

The repository name uses FP8 because FP8 is both the dynamic activation family and the retained secondary weight format. R12 means twelve main routed-expert layers are retained in FP8.

Compatibility

Use the linked recipe for this mixed checkpoint. It provides the ARM64/SM121 runtime, per-layer MXFP4/FP8 loading, NVMe PLE, packed draft head and image limits. An ordinary MXFP4 loader alone is insufficient. The pinned runtime, patches and known limitations are documented in the deployment guide.

The current runtime contract is:

Retained layers

The following zero-based decoder layers keep their main routed experts in block FP8:

19, 20, 21, 23, 28, 32, 33, 40, 41, 43, 44, 45

Every other main routed-expert layer uses the calibrated MXFP4 artifact. The exact machine-readable routing contract is stored in config.json under:

• quantization_config.fp8_routed_layers;
• quantization_config.fp8_mtp_routed_layers; and
• quantization_config.fp8_ple_embeddings.

Size

The indexed weight payload is 143,023,845,370 bytes: 143.024 GB decimal or approximately 133.201 GiB.

The source FP8 checkpoint contains 185.502 GB of indexed tensor payload. This conversion saves 42.478 GB, or 22.90%, without further quantizing the PLE table or the 12 retained routed layers.

Dynamic MXFP8 activations are created at runtime and do not occupy checkpoint storage.

Quantization and layer selection

The conversion started from the released FP8 weights; a BF16 copy of the full model was not required.

The production calibration corpus used the NVIDIA ModelOpt-style source mix:

• 512 CNN/DailyMail samples; and
• 512 samples from nvidia/Nemotron-Post-Training-Dataset-v2.

Real routed activations were captured from the source FP8 model. Candidate MXFP4 expert artifacts were optimized and selected against held-out route-weighted expert outputs. All 48 routed layers were then measured on a separate sensitivity corpus. Whole-model retain-0/5/10/12/14 checkpoints were evaluated on a frozen next-token likelihood split before retain-12 was selected.

Retain-12 had the best point-estimate NLL among those predeclared candidates. Its difference from retain-10 and retain-14 was not statistically decisive; R12 is the frozen selector result, not a claim that twelve retained layers are universally optimal for every workload.

The full conversion provenance is retained in hybrid-conversion-manifest.json.

Accuracy compared with the FP8 source

Frozen selection split

The selection split contains 200 documents and 409,400 matched next-token positions. MTP and CUDA graphs were disabled for deterministic comparison.

Retain-12's selection-split perplexity increase is 0.0046089133, or 0.104223%.

Untouched confirmation split

The final corpus was not used to select the retained-layer count. It contains 256 documents and 524,032 target positions.

The paired relative-NLL 95% bootstrap interval was [+0.075775%, +0.181898%]. The accuracy difference is small but measurable and must not be described as lossless.

Full-vocabulary distribution comparison

Across 512 held-out contexts:

Behavioral diagnostic

The FP8 source passed 5/5 frozen DeepSeek Harness tasks and retain-12 passed 4/5. Retain-12 passed exact response, Python repair, structured JSON, and surgical editing. The one failure preserved the correct state and answer but added explanatory prose where the strict validator required an exact two-word reply. The harness adapter omitted explicit sampling controls, so this isolated sampled difference cannot be causally attributed to quantization. It remains disclosed rather than hidden.

Single-Spark runtime

The recipe keeps the approximately 51.2-GB PLE table on local NVMe and fetches the required rows instead of loading the complete table into shared memory. The remaining indexed weight payload is approximately 91.824 GB; runtime buffers, KV cache and draft-head storage are additional.

Decode results for coding, summarization and creative writing, plus 32K prefill at different context depths, are in the GitHub benchmark tables.

License and attribution

The converted weights remain subject to the included Qwen Community License. See LICENSE. This repository does not change or replace the source model's license terms.

Source model and architecture credit belongs to the Qwen team. This repository contains a third-party mixed-precision conversion and should not be interpreted as an official Qwen release.