Qwen3.8-Flash-Next-FP6-INT8

Qwen3.8-Flash-Next FP6-INT8

Qwen3.8-Flash-Next (180 B parameters including the 51 B n-gram/PLE embedding table) quantized for 2×64 GB GPUs: fp6 e2m3 routed experts, int8 dense projections, bf16 remainder, and the n-gram table as an fp8 sidecar served from pinned host RAM. Full 262,144-token context; KV cache, activations and recurrent state are not quantized.

Provenance: quantized from the official FP8 release (Qwen/Qwen3.8-Flash-Next-FP8, fine-grained fp8, block size 128), not from bf16 weights. Tensors listed as bf16 hold values inherited from that checkpoint. The n-gram table is fp8 e4m3 in the official release; the sidecar re-lays it out without changing values.

This is the second generation of the format. The first (v1, the FP6 repository: fp6 experts, bf16 everything else) remains available; this repository (v2) adds the int8 tier. v2 is produced from v1 with tools/int8_encode.py (CPU, ≈15 min); w8_manifest.json lists every int8 tensor with shape and measured error. The serving stack in patches/ loads either and keys on quantization_config.dense_int8.

Why this build

The previous generation (v1 with its 2026-08 launcher: mem-fraction 0.945–0.955, chunked prefill 8,192, GPU image preprocessing, no input-logprob limit) crashed intermittently under agent traffic. Each cause was isolated and closed (STABILITY.md): MMU faults at the top of VRAM during long prefills, a poisoned CUDA context in the tokenizer process after large images, and out-of-memory on input-logprob requests. This build has run without a crash since those fixes, decodes 61 tok/s single stream against 54–56, and holds a 45 % larger KV pool. As of 2026-09-03 it is the fastest known public build of this model for sm80/CMP-class hardware.

Spec sheet

Hardware profile

Measured (2026-09-02, this checkpoint, reference configuration)

Step budget and hardware floors

Decode step at batch 1: ≈30.6 ms per card. Components:

Prefill is interconnect-bound on this hardware: 101 all-reduces × 21 MB per 4,096-token chunk over ≈1.2 GB/s is ≈2 s of each ≈3.2 s chunk, a ceiling of ≈2,000 tok/s at TP=2 on a gen2 x4 link. The CMP 170HX is limited to PCIe gen2; the same stack on cards with a PCIe 4.0 x16 link (≈25 GB/s practical, e.g. A100 PCIe) cuts the bus time per chunk to ≈0.1 s. Estimated chunk time then ≈1.3 s, ≈3,000 tok/s, bounded next by the expert GEMM. Per added decode stream the step grows ≈15 ms: ≈6 ms expert GEMV (rows), ≈9 ms unattributed (suspects: per-token sparse-attention verify, per-sequence graph nodes).

A 20–24 ms step (85–110 tok/s single stream at the present accept length) is reachable with known techniques. The figures here represent about one week of kernel work and are the limit of this release, not of the hardware. Contributions that fuse the attention/GDN chain, improve the all-reduce schedule, or batch the sparse-attention verify path are welcome; measurements taken with the harnesses in tools/ pair directly with the tables above and can be reported in the repository discussions.

Precision map

From the safetensors headers. Parameters in billions; rel-RMS is the relative RMS error of the stored weights against the FP8-derived bf16 source.

Totals: 128.8 B parameters in shards + 51.2 B in the sidecar. v2 ≈ 6.5 bits/parameter (shards), ≈ 6.9 overall. KV cache bf16, GDN state fp32, activations bf16, accumulation fp32.

Selection rule for the int8 tier: dense projections with large input dimension and well-conditioned ranges, whose int8-g32 error (0.5–0.9 %) is below the expert error (2.44 %). Kept in bf16: routers and the QSA indexer (top-k selection boundaries; a 0.5 % logit perturbation changes routing on near-ties), GDN in-projections and gates (feed a recurrent state that integrates error over the sequence), hyper-connection mixers (scale the residual stream across 48 layers), norms, vision tower, MTP draft layers. QSA q/k/v are a single fused qkv_proj parameter in the serving model and are quantized together. Format details: FORMAT.md; procedure: QUANTIZING.md.

Quantization quality

All checks paired: same prompts, seeds, temperature 0 and server settings on both sides. Decoding is not byte-reproducible at temperature 0 (see Known limits), so single greedy diffs are not used as evidence.

Kernel level:

• fp6 decode bit-exact for all 64 e2m3 codes in the tiled, GEMV2 and GEMV3 kernels; bf16 cast of every decoded value exact.
• GEMV3 vs fp64: rel-RMS 1.66e-3 (bf16 output rounding), equal to GEMV2; element-wise ≤2 bf16 ulps from GEMV2 on the per-card shard shapes with real expert planes (layers 0/7/23/47, both cards); 224 adversarial cases per card (plane patterns × fp16 scale extremes × activation extremes), 0 failures, no inf/nan where fp64 is finite.
• int8 kernel: every code −127…127 in every group position × scale extremes × activation extremes, GEMV (M ≤ 16), dequant to cuBLAS (M > 16) and embedding paths vs fp64: 1,730 checks, 0 failures (75 skipped at fp32 overflow).

Model level:

• Per-layer output error of the int8 layers vs bf16 on real activations: 0.6–0.8 %.
• Teacher-forced log-probabilities, 6,131-token code/documentation corpus, 512-token windows: perplexity 1.6337 (v2). v1 side not yet captured.
• Long-context recall: one planted code in random-word haystacks of 33k / 104k / 209k tokens at depths 0.1 / 0.5 / 0.9: 9/9 (v2), with two concurrent requests, prefill 1,050–1,110 tok/s.
• Deterministic smoke set (arithmetic, bat-and-ball, capital/year, iterative code, list recall, prose): identical correct answers on v1, v2, v2+GEMV3.

Benchmarks

Served by this checkpoint on the hardware above, thinking mode, temperature 0 (the official protocol samples at T = 1.0, top-p 0.95, top-k 20).

GPQA Diamond (198 questions, simple-evals prompt, single greedy run)

Paired:

• v2 vs v1, 71 common questions: 68 vs 66; same letter 68/71; v2 alone correct on 2, v1 alone on 0, both wrong with different letters on 1. Within binomial noise (±3 %) at n = 71.
• v2 at 262k vs 65k budget, 79 questions: 75 vs 70; same letter 72/79; 262k alone correct on 6 (all 3 questions truncated at 65k included), 65k alone on 1. Completion tokens at 262k: median 5.5k, mean 11.5k, max 76k.
• Control: Qwen3.8-27B (dense, unquantized, same harness, 65k budget): 173/198 = 87.4 % (official 89.2), 8 truncations; on the 45-question subset 39/45 vs Flash-Next-FP6 38/45, with 4 wrong completed answers vs 0.

SWE-bench Pro

Official reference 62.5. Seeded 50-task subset, mini-swe-agent 2.4.6, temperature 0, 16,384 max tokens per turn, 75 steps, 40-min wall clock per task, local Docker evaluation: 9/9 resolved (7 Python, 2 JavaScript; mean 40 steps, 14.5 min per task) on v2 + GEMV2, stopped there to keep the baseline on one kernel. Small n; establishes multi-file repository work through tool calls, not a leaderboard number.

Serving

Reference stack: SGLang fork (branch fp6-stable, 16 files changed against upstream 7c66045d71; diff in patches/sglang-fp6.patch), Triton 3.7.1, torch 2.13 / CUDA 13.3, NCCL 2.29, 2× 64 GB sm80 GPUs at TP=2, ≈50 GiB pinned host RAM for the sidecar (cudaHostAlloc, not /dev/shm; 64 GiB system RAM minimum). All kernels are Triton with integer bit-manipulation decode; no fp8 tensor cores or Hopper/Blackwell intrinsics. sm89/sm90/sm120 untested.

                 ┌───────────────────────────────────┐
                 │ Disk (≈145 GiB)                   │
                 │ 25 safetensors shards (≈98 GiB)   │
                 │ ple/ngram.bin       (≈48 GiB)     │
                 └─────────────────┬─────────────────┘
                                   │ read once at startup (≈9 min incl. graph capture)
          ┌────────────────────────┴────────────────────────┐
          ▼                                                 ▼
┌───────────────────────┐  101 all-reduces / step   ┌───────────────────────┐
│ GPU 0 (64 GB, ≈61 in  │◄════════ PCIe ══════════► │ GPU 1 (64 GB, ≈61 in  │
│ use)                  │   (TP=2: each GPU holds   │ use)                  │
│  fp6 experts (half)   │    half of every tensor)  │  fp6 experts (half)   │
│  int8 dense (half)    │                           │  int8 dense (half)    │
│  bf16 GDN/hc/vision   │                           │  bf16 GDN/hc/vision   │
│  MTP draft head       │                           │  MTP draft head       │
│  KV cache, bf16       │  425,024-token pool       │  KV cache, bf16       │
│  GDN state, fp32      │  (24 mamba slots)         │  GDN state, fp32      │
│  decode CUDA graphs   │                           │  decode CUDA graphs   │
└──────────┬────────────┘                           └──────────┬────────────┘
           │  zero-copy gather, ≈2.5 KB/token (16 rows × 160 B)│
           ▼                                                   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Host RAM: n-gram / PLE table, fp8, 47.7 GiB, cudaHostAlloc-pinned       │
│ (vocab-sharded across the two ranks under TP; never occupies VRAM)      │
└─────────────────────────────────────────────────────────────────────────┘

The n-gram table is an embedding lookup, never a matmul operand: ≈16 rows (2.5 KB) per token are gathered from pinned host memory over PCIe. Compute weights stay resident in VRAM.

Reference configuration (launch-example.sh)

Decode speed by change

Details, methodology and negative results: OPTIMIZATION-REPORT.md.

Stability

Four failure modes found under agent traffic and closed; incident timelines, root causes and verification in STABILITY.md.

1. Input-logprob requests over long prompts crashed the server. A /generate request with return_logprob and logprob_start_len=0 on a ≈5k-token prompt (the shape of echo=True, logprobs=N) materializes [positions × vocab/TP] fp32 logits per prefill chunk: 2.0 GB per tensor per card at chunk 4,096 with the 248k vocabulary, plus the log-softmax, into ≈0.8 GB of headroom. Both TP ranks died. Fix: a cap on scored prompt positions per request, SGLANG_MAX_INPUT_LOGPROB_TOKENS (default 1,024), enforced in the tokenizer manager as HTTP 400 with a message stating the limit and the windowed alternative, and again in the scheduler as an abort for callers that bypass validation. Verified after relaunch: 5,000 positions gives 400 with the server up; 500 positions gives 499 logprobs. 1,024 positions cost ≈1 GB per card, a ≈4× margin below the ≈4,000 that exhaust an idle card. tools/tf_logprobs.py scores long texts in 512-token windows over the prefix cache.
2. MMU faults (Xid 31) during long prefills: the allocator grows ≈3.4 GB at chunk 8,192 (≈1.7 GB at 4,096) with context; on these cards a first-touch write into the top ≈140 MiB of VRAM faults instead of raising OOM. Reproduced with a standalone allocation. Rule: ≥1 GB free at steady state after a long prefill, hence chunk 4,096 and mem-fraction 0.93; validated with a 209k-token prefill.
3. Image requests failing after one large image while text served: the fast image processor runs on the GPU in the tokenizer process, which has no memory-budget share; its context was poisoned by an Xid 13. --image-processor-backend pil; the tokenizer process now holds no VRAM; 16.7 MP images pass.
4. Host-RAM livelock during weight loading on a 64 GiB box (48 GiB pinned table + 100 GiB streaming through the page cache). Launcher cgroup cap (MemoryHigh=54G, MemoryMax=57G) and a MemAvailable ≥ 52 GiB gate.

Known limits

• Temperature-0 decoding is not byte-reproducible run to run: decode logits carry allocator-layout-dependent noise (KV page and GDN slot assignment) that flips tokens where the top-2 gap is small (≈1 % of steps below 0.01 nats). Two runs of the same kernel diverged at the same token as a GEMV3-vs-GEMV2 comparison (gap 0.125 nats). No outcome-level effect measured (30/30 repeated greedy runs on a reasoning battery; paired GPQA flips symmetric). All custom kernels verified bitwise deterministic in isolation. Consequence: compare configurations with paired benchmarks and element-wise kernel tests, not text diffs.
• Concurrency ceiling 3 with speculation (GEMV row cutoff); 6 without.
• Speculation and pipeline parallelism are mutually exclusive in SGLang; PP works with the patches (output-identical) but is slower than TP=2 on PCIe-only hosts.
• Prefill CUDA graphs cannot be captured on sm80; the torch profiler stalls the scheduler; CUPTI is unavailable on CMP dies. All timings are CUDA events and in-graph microbenchmarks.
• A prefilling stream pauses the decoding streams (no mixed chunking in the reference configuration).

Repository contents

• 25 safetensors shards (97.6 GiB) + model.safetensors.index.json, config.json (fp6 quantization_config with the dense_int8 block), w8_manifest.json, tokenizer files, chat template, generation and preprocessor configs.
• ple/ngram.bin + ple/ngram-manifest.json: the n-gram sidecar (byte-identical to v1).
• patches/: sglang-fp6.patch (the fork as one diff against upstream 7c66045d71) and the new files whole: fp6.py (quant method, MoE dispatch, int8 wiring), fp6_moe_kernels.py (tiled, GEMV2, GEMV3, split-K fp6 kernels), w8_dense.py (int8 skinny GEMM, dequant path, embedding gather), qsa_packed_decode.py (Triton packed-decode attention). The diff also carries the input-logprob guards, the sidecar loader, qwen4_exp pipeline-parallel support, the QSA indexer sync fix, CPU image decode and env-gated phase timers.
• tools/: encoders (fp6_encode.py, int8_encode.py, ple_extract.py), code-selection probe, fp6 kernel correctness battery and soak, GEMV3 and int8 full-range and real-tensor tests, input-logprob guard unit test, and the measurement harnesses behind every number above (decode_bench.py, prefill_bench.py, concurrency_bench.py, needle_test.py, tf_logprobs.py, smoke_prompts.py, gpqa_diamond_eval.py).
• launch-example.sh: reference launcher, parameterized.
• OPTIMIZATION-REPORT.md, STABILITY.md, QUANTIZING.md, FORMAT.md, LICENSE (Qwen Community License 1.0, inherited from the base model).