Ling-3.0-flash-NVFP4

Ling-3.0-flash — NVFP4 (safetensors)

[!CAUTION]
THIS repo is broken! We're working on fix, sorry!

inclusionAI/Ling-3.0-flash quantized to NVFP4 in the compressed-tensors format, for vLLM and TensorRT-LLM.

76.9 GB across 16 shards. The GGUF builds for llama.cpp live in AtomicChat/Ling-3.0-flash-NVFP4-GGUF.

Mean KL divergence against the bf16 baseline: 0.0409, top-1 agreement 95.9% - more accurate than our GGUF NVFP4 at a comparable size. Method and full numbers below.

What this is

Scheme NVFP4A16, as written into config.json:

format: nvfp4-pack-quantized | producer: compressed-tensors
group_0: bits=4  type=float  strategy=tensor_group  group=16  symmetric=True
targets: ['Linear']   activations: None

4-bit E2M1 weights, one UE4M3 scale per 16 weights, plus a per-tensor global scale — tensor_group is exactly the part the GGUF block layout has no room for, and it does put this build closer to the original than our GGUF NVFP4 at the same bit width. Activations stay bf16, so no calibration data is involved.

Same layout policy as the rest of our line: only the routed experts, 97.1% of the weights, are quantized. Everything cheap in parameters but expensive in error is left alone:

QuantizationModifier(
    targets=["Linear"],
    scheme="NVFP4A16",
    ignore=[
        "lm_head",
        r"re:.*mlp\.gate$",          # the router: an error changes which expert runs
        r"re:.*shared_experts\..*",  # seen by every token
        r"re:.*attention\..*",       # MLA, KDA projections and gates
    ],
)

In the written config those patterns are expanded into an explicit list of more than 400 modules: every attention.* projection (the KDA f_proj, b_proj, g_proj and the MLA kv_a_proj_with_mqa, kv_b_proj, dense), every shared_experts.* and lm_head. Only the routed experts are in NVFP4.

The MTP block (layer 42) is kept here in bf16. The GGUF converter drops it, so if multi-token prediction lands in vLLM, this build has what it needs and the GGUF one does not.

Running it

Ling 3.0 is not in upstream vLLM yet, but the support is open as PR #51045 and can be installed straight from it, no need to wait for the merge:

git clone https://github.com/vllm-project/vllm && cd vllm
gh pr checkout 51045          # or: git fetch origin pull/51045/head:ling3 && git checkout ling3
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

vllm serve AtomicChat/Ling-3.0-flash-NVFP4 --trust-remote-code --tensor-parallel-size 4

The model authors also publish a fork, inclusionAI/vllm-ling-v3, which works as well; the PR is closer to what will eventually ship upstream.

On arm64 machines such as DGX Spark, VLLM_USE_PRECOMPILED=1 will not help - those wheels are built for x86_64 - so expect a full source build there.

NVFP4 has native tensor-core support on Blackwell (sm_100, sm_120, sm_121). Elsewhere it runs through dequantization, without the speedup.

Measuring it

NVFP4A16 is weight-only: activations stay bf16, so a model whose weights have been round-tripped through the FP4 grid is mathematically the same model as this compressed checkpoint. That makes the measurement easy to keep honest - the checkpoint is expanded back to dense bf16 and then goes through the exact same harness as every other rung: llama-perplexity --kl-divergence against the same bf16 baseline, the same 92 chunks of held-out text at 4096 tokens, the same 4x RTX PRO 6000 machine. No second definition of the metric, no cross-runtime fudge factor.

The expansion was checked against compressed_tensors' own unpack_fp4_from_uint8 before use (bit-identical) and against the original bf16 weights (relative L2 error 0.095 on the first tensor, which is what FP4 costs).

Distribution of the divergence: median 0.0015, 99th percentile 0.448, maximum 21.9. As with every 4-bit build, the mean is carried by a small tail of tokens where the router flips to a different expert, not by a uniform blur.

Two things separate this build from the GGUF NVFP4 of the same width, and they pull in opposite directions. In its favour, tensor_group carries a per-tensor global scale that the GGUF block has no room for, and the ignore list keeps attention and the shared experts in bf16 rather than Q8_0. Against it, the two dense FFN layers at the front of the model land in NVFP4 here, where the GGUF build keeps them at Q8_0. On net it comes out 0.013 ahead.

Worth stating plainly: at 4-bit, NVFP4 is not the accuracy-per-byte winner in this line. AD-IQ4_XXS is both smaller and closer to the original. NVFP4 is here for the Blackwell tensor cores, not for the error curve.

Raw logs and json: AtomicChat/Ling-3.0-flash-GGUF-metrics.