Ling-3.0-flash-NVFP4

Ling-3.0-flash — NVFP4 (weight-only, mixed precision)

A 4-bit NVFP4 quantization of Ling-3.0-flash, produced with qstream. The routed MoE experts (~95% of the weights) are quantized to NVFP4; everything quality-sensitive stays BF16.

The original model card follows in full below.

Speculative decoding (MTP) needs specific flags and a vLLM patch

It works well — 56.5% acceptance, 2.70 tokens per decode step (94.3% at draft position 0) — but NVFP4 needs more setup than the MXFP4 build does:

--kernel-config '{"moe_backend":"marlin"}'   --max-num-seqs 256

plus the unquantized.py patch in vllm_patch/. Why each is needed:

• marlin is the only MoE backend supporting weight-only NVFP4A16. triton is rejected outright; cutlass and flashinfer_cutlass reject the scheme (QuantKey(u8,scale(f8e4m3fn,...))).
• The patch exists because moe_backend is a single global setting, while this checkpoint has two MoE kinds: NVFP4 routed experts (need marlin) and the BF16 MTP layer (marlin has no unquantized kernel → moe_backend='marlin' is not supported for unquantized MoE). The patch routes unquantized MoE to Triton, mirroring the existing humming precedent. Triton also avoids FlashInfer TRT-LLM's grouped-routing kernel, which is warp-limited to 32 experts per group — Ling-3.0 has 64 (512 experts / 8 groups) and either crashes there or silently mis-routes.
• --max-num-seqs 256 — the BF16 MTP head takes ~6 GB, shrinking the KDA state cache below the default 1024 sequences, so CUDA graph capture aborts with max_num_seqs (1024) exceeds available Mamba cache blocks (684).

Acceptance by draft position (conditional): 94.3% / 57.2% / 39.7%. The falloff is expected — Ling-3.0 has a single MTP layer re-run per speculative token — so num_speculative_tokens: 2 may beat 3 on net throughput. MTP also roughly doubles cold start: the draft loader re-reads the whole checkpoint to extract one layer.

With all of the above, graph capture succeeds and --enforce-eager is not needed. If you served this checkpoint before 2026-08-05, re-pull config.json — the earlier one let the layer-agnostic targets regex claim the MTP layer's BF16 experts as NVFP4, giving a silently broken drafter (0% acceptance, no error).

Which build should you use?

Most people should take the MXFP4 build instead. NVFP4 reconstructs the weights substantially more faithfully, and that advantage does not show up downstream:

Both GSM8K gaps are under 1.2σ — the two builds are statistically indistinguishable on this benchmark despite NVFP4's much lower weight error. At this magnitude, 4-bit reconstruction error is already below what GSM8K can resolve. NVFP4 is published for comparison and for anyone wanting the more faithful weights (e.g. for longer-generation or harder tasks where the difference may still surface); MXFP4 is the smaller, better- tested release.

What is quantized to what

61,440 tensors quantized (40 layers × 512 experts × 3 projections), each with a packed weight, an FP8-E4M3 group scale and a per-tensor global scale.

Fidelity

Weight-reconstruction relative error, 960-tensor stratified sample (24 per layer, expert ids strided across the full 512 range, all 40 layers):

NVFP4's worst sampled tensor (0.0867) is better than MXFP4's best (0.1099) — the two distributions are disjoint. Two effects compound: group 16 rather than 32 halves how many weights share a scale, and FP8-E4M3 scales are continuous where MXFP4's E8M0 scales are powers of two, so an MXFP4 block whose ideal scale falls between 2^k and 2^(k+1) must round and waste up to 2× of range.

Known issues

1. MTP / speculative decoding: 0% acceptance — see the warning above. Serve without --speculative-config.
2. MTP also breaks CUDA graph capture (gdn_attn.py:382 → CUDA error: invalid argument). Without a draft model, capture succeeds and --enforce-eager is not needed.
3. The MTP draft loader re-reads the entire checkpoint to extract one layer — two full weight-loading passes of near-identical duration at startup. Costs I/O, not VRAM.
4. NVFP4 needs qstream ≥ the _prune_ignore fix. Earlier qstream generalised ignore entries by layer index, so excluding the MTP layer emitted re:.*layers\.\d+\.mlp\.experts\.\d+\.gate_proj$ — which also ignored the quantized experts in all 40 other layers. vLLM checks should_ignore_layer before target matching, so every expert silently resolved to unquantized and the loader tried to allocate BF16 buffers for ~242 GB of weights (OOM with no useful error). The config.json in this repo is already correct.

Serving with vLLM

Ling-3.0 (BailingMoeV3ForCausalLM, model_type: bailing_hybrid) needs the vendor fork inclusionAI/vllm, branch ling_3_0. (The base model card points at inclusionAI/vllm-ling-v3, which does not exist.) Upstream vLLM has only BailingMoe / V2 / V2.5.

Required: the MoE clamp patch

Ling-3.0 declares a per-layer clamped SwiGLU (expert_swiglu_limit_list = [0]*35 + [4]*7, so layers 35–41 clamp at 4). vLLM's compressed-tensors MXFP4/NVFP4 MoE path drops that clamp: the quant method never reads layer.swiglu_limit, the quant-config builder has no parameter to receive it, and run_cutlass_moe_mxfp4 calls apply_moe_activation() with no clamp argument. The result is unclamped SwiGLU on layers 35–41 — the divergence is large, not marginal (gate=10 gives silu(10)*2 = 20.0 versus silu(4)*2 = 7.84).

Apply the three files in vllm_patch/ over the fork before serving; they are pure Python, so an editable install needs no rebuild. See vllm_patch/README.md.

Launch (Docker)

A prebuilt runtime image carrying the fork, both patches and flashinfer-jit-cache. NVFP4 + MTP requires the nvfp4 tag or newer latest — earlier images lack the unquantized MoE oracle patch and will fail to start with moe_backend='marlin' is not supported for unquantized MoE. (The image itself is model-agnostic and serves both builds; the tag just marks the build that added this.)

Weights are pulled from the Hub, so mount a cache directory on a volume with ≥85 GB free (without it the download lands in the container's writable layer and will fill your root filesystem):

docker run --gpus all -d --name ling3 --ipc=host -p 8000:8000 \
  -e HF_HOME=/hf -v /path/with/85GB/free:/hf \
  olkafi/vllm-bailing-v3:nvfp4 \
  olka-fi/Ling-3.0-flash-NVFP4 \
  --served-model-name ling3 --trust-remote-code \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.85 --max-model-len 16384 --max-num-seqs 256 \
  --enable-prefix-caching --mamba-cache-mode align \
  --kernel-config '{"moe_backend":"marlin"}' \
  --enable-auto-tool-choice --tool-call-parser ling3 --reasoning-parser ling3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Drop --speculative-config to serve without MTP; --kernel-config and --max-num-seqs 256 are then optional, and --max-model-len can go much higher (32K+). Add --load-format fastsafetensors for faster weight loading.

--enable-prefix-caching is required for --mamba-cache-mode align to take effect — without it vLLM silently downgrades the KDA linear-attention state cache to none and only logs a warning.

Recommended sampling (from the base card): temperature=0.6, top_p=0.95, top_k=20, with enable_thinking.

Provenance

Built with qstream from the BF16 inclusionAI/Ling-3.0-flash release:

qstream-quantize \
  --model_dir  Ling-3.0-flash \
  --output_dir Ling-3.0-flash-NVFP4 \
  --quant_format nvfp4 --nvfp4_mse --format ct \
  --include_layers '*.mlp.experts.*' \
  --exclude_layers '*.layers.42.*' \
  --workers 8 --device cuda

--nvfp4_mse searches neighbouring E4M3 group scales and keeps the minimum-reconstruction- error one, with γ-weighted activation awareness from post_attention_layernorm.

Evaluation notes

GSM8K figures above are 5-shot over the full 1319-problem test set via /v1/completions with plain few-shot prompting — a non-thinking protocol. They are not comparable to the base model's published scores, which are generated with thinking mode + CoT. They exist to bound quantization damage, not as leaderboard claims.

License

Inherits the MIT license from the base model. This is a derivative (quantized) work of inclusionAI/Ling-3.0-flash.

---

Original model card

<p align="center"> <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100"/> </p> <p align="center">🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>&nbsp;&nbsp; | &nbsp;&nbsp;🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope </a>&nbsp;&nbsp; | &nbsp;&nbsp;🐙 <a href="https://openrouter.ai/inclusionai/ling-3.0-flash:free">OpenRouter </a>&nbsp;&nbsp;</p>

<font style="color:rgba(0, 0, 0, 0.88);">Introduction</font>

We're introducing Ling-3.0-flash, our next-generation native hybrid reasoning model. Operating with 124B total and 5.1B active parameters (~12.4% and ~8.1% of our previous 1T-class flagship Ring-2.6-1T), Ling-3.0-flash matches or outperforms its predecessor across key benchmarks.

<font style="color:rgb(16, 18, 20);">Key highlights of the model are summarized below:</font>

• Native Hybrid-Linear Architecture: Ling-3.0 adopts a native hybrid linear attention architecture from the very start of pretraining (5:1 alternating stacking of Kimi Delta Attention (KDA) and MLA), upgraded with KDA fine-grained diagonal gating and 1/64 sparse MoE. With 124B total parameters and 5.1B activated parameters, it achieves a synergistic leap in long-context efficiency and computational cost.
• Remarkable Efficiency & Performance: Engineered for speed, compute efficiency, and production deployment, Ling-3.0-flash delivers class-defying performance against both larger SOTA competitors and previous-generation flagships. Activating only 5.1B parameters per token, it provides impressive reasoning, instruction following, and long-context capabilities to empower complex agentic workflows in production environments.
• Comprehensive Agentic Evolution: Tailored for real-world productivity workflows, the model incorporates 10,000+ interactive training environments to achieve end-to-end closed-loop execution across Coding, General, and Deep Research Agent tasks. It natively integrates the SGLang HiCache + Mooncake hierarchical caching architecture (featuring physical dual-pools and a cluster-shared L3 cache), eliminating redundant recomputation during long-horizon interactions and reducing Time to First Token (TTFT) by 60% to over 80% in long-input scenarios.s the SGLang HiCache + Mooncake hierarchical caching architecture (featuring physical dual-pools and a cluster-shared L3 cache), eliminating redundant recomputation during long-horizon interactions and reducing Time to First Token (TTFT) by 60% to over 80% in long-input scenarios.

<!-- 这是一张图片，ocr 内容为：SWE-BENCH MULTILINGUAL TERMINAL-BENCH 2.1 SWE-BENCHPRO TAU3-BANKING-AA LL088000LL 65 6 71.2 76.5 75.9 AP 28.0 60 0 72.4 73.3 56.6 56.2 56.3 71.0 53.9 70 22.9 55 57.0 50 47.9 48.3 60 56.7 14.6 45 11.3 71.3 39.3 39.0 70.1 50 8.9 40 42.7 40 34.1 35 30 30 WIDESEARCH MCP-ATLAS SKILLSBENCH BROWSECOMP 品质88品质品等学导新品 品民品品导品只只口 豆豆复品品导导导 品品&品品定品名品 6 4 79.5 米 0 75.2 73.6 74.4 70.2 69.0 82.0 66.7 6 65.5 62.2 53.5 75.8 61.2 73.2 71.7 74.0 55.2 53.6 52.6 49.4 G 20.3 19.5 71.9 31.3 MULTI-AGENT IFBENCH SYSBENCH MRCR-256K MULTI-IF 100 雪饼 110 110 87.7 89.3 79.2 06 82.9 84.6 86.2 84.6 84.8 75.7 82.3 93.6 91.4 93.9 90793.3949 72.6 84.3 90 品8888元 81.1 86.5 86.2 69.0 70 67.3 70 60 56.6 05 30 0 30 10 ULIO RING-2.6-1T(XHIGH) STEP-3.7-FLASH(HIGH) LING-3.0-FLASH MINIMAX-M2.7 WALBAN DEEPSEEK-V4-FLASH-PREVIEW(MAX) CLAUDE-SONNET-4.6(MAX) GPT-5.4-MINI(HIGH) NEMOTRON-3-SUPER-120B-A12B NOTE:THINKING MODE IS ENABLED BY DEFAULT. -->

Model Overview

The model summary information and architecture diagram are as follows:

<!-- 这是一张图片，ocr 内容为：LING-3.0-FLASH ARCHITECTURE TRAININGOBJECTIVE:NEXT-TOKENPRE TOKEN PREDICTION AND MULTI-TOKEN PREDICTION (MTP) VOCABULARY O:SIGMOID GATE OUTPUT LINEAR OUTPUT LAYER SIZE OF 157K PROJECTION FINAL RMSNORM MULTI-HEAD LATENT ATTENTION K E512A8+1SHARED MOE ROPE ROPE ERT,ALF-LB EXPERT RMSNORM LINEAR LINEAR RMSNORM RMSNORM SUPPORTED LINEAR LINEAR LINEAR GATED MLA ROPE CONTENTLENGTH-- OF 1M TOKENS RMSNORM O:SIGMOID GATE OUTPUT P:SOFTPLUS GATE PROJECTION 8:SWISH FUNC RMSNORM MOE KIMI DELTA ATTENTION RMSNORM K V L2 NORM LINEAR TIME 8 KDA COMPLEXITY RMSNORM CONV CONV MLP LINEAR TOKEN EMBEDDING LAYER 7 GROUPS LINEAR LINEAR FIRST 2 BLOCKS EMBEDDING TOKENIZED TEXT USE DENSE FFN DIMENSION OF 2,560 个 INSTEAD OF MOE SAMPLE INPUT TEXT -->

Evaluation

We have conducted a comprehensive evaluation of Ling-3.0-flash across multiple authoritative benchmarks. Ling-3.0-flash performs strongly on representative code/agent benchmarks such as SWE-Bench Pro, SWE-Bench Multilingual, Tau3-banking-AA, MCP-Atlas and SkillsBench, etc. In practice, Ling-3.0-flash delivers a strong user experience across frameworks including Claude Code,Kilo Code,Qwen Code,Hermes Agent,and OpenClaw, etc. Beyond agentic tasks, Ling-3.0-flash also delivers strong performance across general knowledge,mathematical reasoning,instruction following,and long-context understanding.

<!-- 这是一张图片，ocr 内容为：DEEPSEEK CLAUDE- NEMOTRON- LING-3.0 GPT-5.4 STEP-3.7- V4-FLASH- RING-2.6-1T MINIMAX- 3-SUPER FLASH (HIGH) MINI (HIGH) FLASH (XHIGH) 120B-A12B (MAX) (MAX) 120B-A12B 124B-A5.1B 284B-A13B 198B-A11B 230B-A10B 1T-A63B SIZE CODING AGENT SWE-BENCH PRO 47.9 34.1 48.3 56.2 56.3 52.6 53.9 56.6 SWE-BENCH MULTILINGUAL 72.4 71.0 42.7 56.7 724 76.5 73.3 75.9 57.0 TERMINAL-BENCH 2.1 39.3 55.8 55.0 71.2 62.0 43.1 39.0 59.2 77.0 65.1 55.8 ARTIFACTSBENCH 66.8 64.0 51.6 68.7 25.3 19.2 MINIAPPENCH 28.0 14.8 46.3 20.7 58.8 5.8 52.2 ANTSWEBENCH 48.6 46.4 GENERAL AGENT 8.9 11.3 28.0 TAU3-BANKING-AA 11.3 10.1 22.9 14.6 30.5 55.2 53.6 69.0 65.5 MCP-ATLAS 52.6 61.2 49.4 66.7 53.5 24.9 20.3 44.8 54.4 28.4 44.8 11.9 SKILLSBENCH BFCL-V4 59.5 73.0 68.3 73.1 65.4 63.6 60.6 64.8 GDPVAL V2-AA 920 1377 1189 1017 1107 699 1159 SEARCH AGENT WIDESEARCH 75.2 70.2 62.2 73.6 19.5 74.4 56.8 79.5 72.2(W/ CTX) 74.0(W/CTX) BROWSECOMP 73.2 71.7 31.3 75.8 76.3 82.0(MA) 82.1(MA) DRACO 71.3 75.8 61.3 66.8 70.4 INSTRUCTION FOLLOWING 75.7 74.5 IFBENCH 72.6 67.3 79.2 69.0 44.6 56.6 91.4 93.6 93.3 SYSBENCH 86.5 93.9 90.7 86.2 94.9 77.3 71.3 LIFEBENCH 69.2 72.5 71.8 74.1 66.9 60.2 REASONING 91.7 94.2 AIME26 93.2 95.0 95.8 92.9 96.5 94.4 HMMT-FEB26 87.0 71.9 93.5 85.6 94.8 87.9 84.9 83.9 83.7 74.5 87.0 IMO-ANSWERBENCH 77.0 82.1 86.1 66.9 22.7 HLE 19.9 20.6 18.3 18.3 34.8 28.1 30.0 LIVECODEBENCH 82.8 78.1 80.8 87.0 75.6 83.7 78.7 91.6 (2408-2505) LONG CONTEXT & MULTI-TURN DIALOGUE 39.2 MRCR 128K 27.7 90.1 56.1 40.8 90.8 92.5 88.5 92.7 81.1 MRCR 256K 25.7 76.5 50.5 35.8 84.3 70.7 AA-LCR 65.1 64.3 68.7 63.7 63.0 58.3 63.4 89.3 MULTI-IF 86.2 87.7 82.9 84.6 84.8 82.3 84.6 -->

• Thinking mode is enabled by default. Unless otherwise specified, the default parameters for Ling-3.0-flash are as follows: temperature=0.6, top_p=0.95, top_k=20.
• <font style="color:rgb(16, 18, 20);">SWE-Bench Series:Evaluated using OpenHands as the agent harness with tailored prompts. Decoding uses </font>temperature=0.6, top_p=0.95, max_new_tokens=32K<font style="color:rgb(16, 18, 20);">, with a 256K context window.</font>
• <font style="color:rgb(16, 18, 20);">Terminal-Bench 2.1: Evaluated under the Artificial Analysis (AA) protocol using the default Terminus 2 harness, a unified 2-hour timeout, the provided JSON parser in preserve-thinking mode, and 3 runs per task (mean). Decoding uses </font>temperature=0.6, top_p=1.0, max_new_tokens=32K<font style="color:rgb(16, 18, 20);">, with a 256K context window.</font>
• <font style="color:rgb(16, 18, 20);">MiniAppBench: A 500-task coding benchmark evaluating whether models can turn a single user request into complete, usable interactive HTML apps in real-world application-generation scenarios. Evaluated with temperature=1.0, top_p=1.0, max_tokens=128K.
• <font style="color:rgb(16, 18, 20);">AntSWEBench: </font><font style="color:rgb(0, 0, 0);">AntSWEBench is an internally used software engineering benchmark that covers mainstream programming languages such as Java, JavaScript, and Python, including various development scenarios like new feature, bug fix, and code refactoring.</font>
• <font style="color:rgb(16, 18, 20);">Tau3-banking-AA: Aligned with the AA leaderboard, utilizing GPT-5.4-mini (medium reasoning) for both the user simulator and the natural-language assertion judge.</font>
• MCP-Atlas: Evaluated on the 500-task public set using the official v1 harness with a 20-turn limit and Gemini-2.5-Pro as the claim-coverage judger.
• SkillsBench: Evaluated via kilo-code on 87 tasks (excluding external API-dependent tasks), averaged over 3 runs.
• <font style="color:rgb(16, 18, 20);">GDPval v2-AA : Evaluated on the public 220-task benchmark using the official Stirrup harness, with a 250-turn limit and a 5-hour timeout.</font>
• <font style="color:rgb(16, 18, 20);">Search‑agent：For all search‑agent tasks, evaluations are performed using an internal harness. The basic ReAct paradigm is adopted for single-agent evaluation, while a multi-agent setup is employed for BrowseComp. The reported metric is the average pass@1.</font>
  • <font style="color:rgb(16, 18, 20);">WideSearch: Evaluated using the official prompt and the official judge model GPT-4.1 on the corrected version of the dataset.</font>
  • <font style="color:rgb(16, 18, 20);">Draco: Scored based on official rubrics per question, with the final score calculated as the average across all questions using </font>Claude Opus 4.6<font style="color:rgb(16, 18, 20);"> as the scoring model.</font>
  • BrowseComp (Single-Agent): Evaluated using a resume strategy for context management: once the context reaches a 64K-token threshold, the trajectory is summarized, the original history is discarded, and execution is resumed from the summary.
  • BrowseComp (Multi-Agent):<font style="color:rgb(16, 18, 20);"> Evaluated on English and ZH Revised datasets using an internal multi-agent search harness based on SearchSwarm/Tongyi DeepResearch, configured with </font>temperature=0.85, top_p=0.95, max_tokens=8K<font style="color:rgb(16, 18, 20);">, and main/sub-agent context windows of 128K and 64K, respectively.</font>

Quickstart

SGlang

Install our SGLang

pip install uv

uv venv ~/my_ling_env

source ~/my_ling_env/bin/activate

git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git

cd sglang_ling_v3

pip install --upgrade pip

pip install -e "python"

Run Inference

Here is an example to run Ling-3.0-flash with 4 GPUs, where the master node IP is ${MASTER_IP} and server port is ${PORT}:

Server

Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-algorithm NEXTN) for lower latency.

export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=1
export SGLANG_ENABLE_SPEC_V2=1
python -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --dist-init-addr $MASTER_IP:2345 \
    --port $PORT \
    --nnodes 1 \
    --mem-fraction-static 0.8 \
    --max-running-requests 64 \
    --tp-size 4 \
    --chunked-prefill-size 8192 \
    --tool-call-parser ling3 \
    --reasoning-parser ling3 \
    --context-length 262144 \
    --speculative-algorithm NEXTN \
    --max-mamba-cache-size 320 \
    --enable-fp32-lm-head \
    --disable-shared-experts-fusion

Client

We recommend using the sampling parameters temperature=0.6, top_p=0.95, and top_k=20, and enabling enable_thinking for better performance.

curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto",
       "messages": [{"role": "user", "content": "hello!"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 0.6, 
       "top_k": 20,
       "top_p": 0.95
     }'

vLLM

Install our vLLM

pip install uv

uv venv ~/my_ling_env

source ~/my_ling_env/bin/activate

git clone -b ling_3_0 https://github.com/inclusionAI/vllm-ling-v3.git

cd vllm-ling-v3

VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

Run Inference

Here is the example to run Ling-3.0-flash with 4 GPUs, where the server port is ${PORT}:

Server

Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-config) for lower latency.

vllm serve "$MODEL_PATH" \
    --port "$PORT" \
    --trust-remote-code \
    --served-model-name auto \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.85 \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --enable-auto-tool-choice \
    --tool-call-parser ling3 \
    --reasoning-parser ling3 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Client

We recommend using the sampling parameters temperature=0.6, top_p=0.95, and top_k=20, and enabling enable_thinking for better performance.

curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto",
       "messages": [{"role": "user", "content": "hello!"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 0.6, 
       "top_k": 20,
       "top_p": 0.95
     }'