Ling-3.0-flash-NVFP4-SGLang-MTP

Ling-3.0-flash — NVFP4, single GPU, MTP working, SwiGLU clamp fixed

Ling-3.0-flash quantized to NVFP4 so it runs on one 128GB-class GPU under SGLang, with NEXTN multi-token prediction working and the SwiGLU clamp actually applied.

Upstream ships a 4×141GB reference deployment. This runs on one card.

Two things make this different from a plain conversion, and I explain both below so you can reproduce them rather than trust me:

1. The MTP layer is preserved and correctly declared, so speculative decoding works.
2. SGLang silently drops Ling's SwiGLU clamp on the NVFP4 path. Without the fix in sglang_patch/, this model — or any NVFP4 build of it — corrupts roughly two thirds of the code it writes. The weights are fine. The serving stack is what needs patching.

---

⚠️ Read this first: apply sglang_patch/ or you get a broken model

Ling is trained with clamped SwiGLU on its late layers. The official config carries the limits:

expert_swiglu_limit_list        layers 35–41 = 4      (all other layers 0)
share_expert_swiglu_limit_list  layers 34–39 = 5, layers 40–41 = 7

SGLang reads those keys and implements the clamp — but the NVFP4 MoE scheme throws it away:

# sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4_moe.py
gemm1_clamp_limit=None,     # hard-coded

Every other MoE runner in SGLang (triton, marlin, ascend, deep_gemm, mxint4) forwards the real value. Only the NVFP4 scheme discards it. With no clamp, activations in the last seven layers can run away, and a junk token wins a step:

stack.append(c)ulp
high = len(arr) -  Advisory
elif arr[mid] < x Advisory:
seen.add(culp)          # should be seen.add(c)

It looks random and it recovers immediately, which is what makes it easy to misdiagnose as a bad quantization. It is not. It is a missing clamp.

What I measured, before and after the patch

Detection is an AST check for identifiers that are loaded but never bound. Use that, not a non-ASCII scan — the most common junk tokens (ulp, oly, culp, Advisory) are plain ASCII and a charset filter sails straight past them.

The upstream-recommended sampling going from 0/10 to 10/10 is the clearest signal. Before the patch that configuration was the worst one; after it, it is perfect.

Applying it

cd <sglang>/python/sglang/srt
patch -p1 < sglang_patch/01-flashinfer_cutlass.patch
patch -p1 < sglang_patch/02-compressed_tensors_w4a4_nvfp4_moe.patch

Or bind-mount the two patched files over the image. Three edits total:

• layers/moe/moe_runner/flashinfer_cutlass.py — add a swiglu_limit field to FlashInferCutlassMoeQuantInfo and pass it into the fused-MoE call. The CUTLASS kernel already accepts swiglu_limit; the sibling mxfp4 quant info passes it and the NVFP4 one simply never did. Both call the same flashinfer_cutlass_fused_moe.
• .../compressed_tensors_w4a4_nvfp4_moe.py — build the clamp tensor from moe_runner_config.gemm1_clamp_limit (Bailing sets that, not swiglu_limit) and pass it. The same file's TRTLLM branch gets the hard-coded None fixed too.

Verify the clamp is live

The patch logs once per layer. You want to see this:

[swiglu-clamp-patch] cutlass nvfp4 limit=None    <- layers 0–34
[swiglu-clamp-patch] cutlass nvfp4 limit=4       <- layers 35–41

If you never see limit=4, the clamp is not being applied and your output will be corrupted. Check that first before blaming anything else.

Do not use the TRTLLM MoE backend with this model

--moe-runner-backend flashinfer_trtllm cannot run Ling at all. Bailing never sets routing_method_type, and once you supply it (RoutingMethodType.DeepSeekV3 — Ling's routing is sigmoid + expert bias, n_group 8, topk_group 4, top_k 8) the routing kernel still refuses the shape:

Routing kernel expects #experts per group <= warp size (32),
got 512 experts / 8 groups = 64 experts per group

512 experts over 8 groups is 64 per group. That is a compiled CUDA limit, not a config problem. Use the default CUTLASS backend — which is where the clamp fix lives anyway.

---

MTP (NEXTN) — how it works here and what to watch

Ling ships one MTP layer (num_nextn_predict_layers: 1, layer 42). It drafts tokens that the main model then verifies in a single batched pass, so you pay one forward pass for several accepted tokens.

Layer 42 is BF16 in this checkpoint — 0 packed tensors, 1552 plain. It must be excluded from the quantization targets. config.json here puts re:.*layers\.42\..* first in ignore.

This matters more than it sounds. If a blanket re:.*\.experts\..* target claims layer 42's experts are FP4 while the tensors are BF16, the draft head loads under the wrong scheme and drafts garbage. You get:

accept len: 1.00, accept rate: 0.00

and speculation becomes pure overhead — it gets worse the deeper you draft (I measured 0.94× at k=1, 0.80× at k=2, 0.71× at k=3 in that broken state). An acceptance rate of 0.00 means your draft head is broken, not that MTP is a bad idea for this model. Read the acceptance number before you conclude anything about speculative decoding.

With layer 42 loading correctly, acceptance is 0.81–0.85 and k=2 is the optimum:

Deeper is not better. k=3 drafts more but accepts less, and the extra verify work eats the gain.

Speculation is not byte-identical

Under greedy, speculation-off is 5/5 self-reproducible, but NEXTN k=2 matches it byte-for-byte on only 1/5 prompts. The differences I saw were paraphrases, not corruption — floating-point non-associativity in the batched verify pass picking a different token at a near-tie.

With --enable-deterministic-inference that rises to 4/5 at 38.26 tok/s. That mode needs one more edit: raise MIN_DEEPGEMM_DIM in sglang/srt/batch_invariant_ops/batch_invariant_ops.py from 16 to 64, because the guard is N >= MIN_DEEPGEMM_DIM and Ling's N=32 MoE router otherwise dies with DeepGEMM failed for matrix shapes M=1, N=32, K=2560.

---

Quantization layout

f_proj and g_proj are the KDA decay and output gates. I keep them at BF16 — a 4-bit gate feeds an exponential decay whose error compounds along the sequence.

Weights 71.8 GB resident, 76 GB on disk, ~300 s to load.

input_global_scale tensors are included — do not strip them

SGLang's CompressedTensorsW4A4Nvfp4MoE allocates w13_input_global_scale and w2_input_global_scale with torch.empty() and fills them only from the checkpoint. A checkpoint without them loads with no error and no warning, leaves those parameters holding uninitialized memory, and computes g_alphas = (1 / w13_input_global_scale) * w13_weight_scale_2 from garbage. The result is NaN logits and an output of nothing but ! characters.

This repo ships 61,440 of them, set to 1.0 (float32), one beside every expert weight_global_scale. With dynamic: "local" activations the per-block scale is computed at runtime and the kernel applies alpha = 1/(g_a·g_w), so g_a = 1 leaves s_a_block = amax/6, comfortably inside FP8 e4m3 range.

These are uncalibrated. A calibrated activation scale may do better; I have not measured that.

Why W4A4 and not W4A16

SGLang implements exactly one NVFP4 MoE path — _is_fp4a4_nvfp4 → CompressedTensorsW4A4Nvfp4MoE. There is no weight-only branch: get_moe_scheme falls through _is_wNa16_group_channel (INT-typed only) to the w8a8 predicates and raises. A W4A16 checkpoint (input_activations: null) hits AttributeError: 'NoneType' object has no attribute 'num_bits' in _is_static_tensor_w8a8 — those predicates dereference input_quant with no None guard.

So config.json declares input_activations (4-bit, dynamic: "local", tensor_group) to select the W4A4 path. That is why the community W4A16 NVFP4 builds of this model will not serve on SGLang.

Serving

Requires an SGLang build with bailing_moe_v3 support, plus sglang_patch/.

python3 -m sglang.launch_server \
  --model-path <this-repo> --trust-remote-code \
  --host 0.0.0.0 --port 30013 \
  --tp-size 1 --ep-size 1 \
  --mem-fraction-static 0.75 \
  --context-length 32768 \
  --max-running-requests 1 \
  --chunked-prefill-size 2048 \
  --disable-shared-experts-fusion \
  --tool-call-parser glm45 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3

--disable-shared-experts-fusion is required — the shared experts are BF16 while the fused MoE expects FP4.

Do not pass --reasoning-parser. With one active, any request sending enable_thinking: false comes back with content: "" and the answer stranded in reasoning_content: the model emits no <think>…</think> wrapper, so the parser never sees a terminator and files the whole output as reasoning. Agentic harnesses that disable thinking will see empty replies.

Native context is 262144 (rope_scaling: null); 32768 above is just what I run.

Provenance

Quantized from the official BF16 release with llmcompressor 0.12.0.1 / compressed_tensors 0.17.1 via model_free_ptq, scheme NVFP4A16.

Sequential oneshot was not usable — compressed_tensors' from_accelerate path asserts on disk/meta tensors on this hardware. The Bailing MoE layout also requires the fused expert maps to be emptied, with MLA kept out through the ignore list instead.

Counts after quantization: 61,440 quantized expert tensors; 16 MLA tensors and all 1,552 layer-42 tensors left unpacked at BF16.

Credit

All credit to InclusionAI for Ling-3.0-flash. The SwiGLU clamp behaviour was first documented for the GGUF conversion path by raulvidis/Ling-3.0-flash-ROCmFP4-STRIX-MTP-GGUF; the fix here is the equivalent for SGLang's NVFP4 CUTLASS path.

License

MIT, inherited from inclusionAI/Ling-3.0-flash.

<!-- PEER-TABLE:START -->

Other public builds of this model

Compiled from Hugging Face repository metadata — file sizes, shipped files, quant variant as named by each repo. No third-party build was run or benchmarked here, so this table makes no speed or quality claim about any of them. It is here so you can see the size and format options at a glance and pick what fits your hardware.

Base model: inclusionAI/Ling-3.0-flash. Generated from Hub metadata; download counts move over time.

<!-- PEER-TABLE:END --> <!-- CREDITS:START -->

Acknowledgements

Base model authors — see base_model in the metadata above; all model weights, licences and capabilities are theirs. This repository contributes quantisation and measurement only.

llama.cpp / vLLM and NVIDIA's NVFP4 tooling — the formats and runtimes these builds target.

<!-- CREDITS:END -->