Ling-3.0-flash-GGUF

Ling-3.0-flash-GGUF — sm_120-safe quants

Quants of inclusionAI/Ling-3.0-flash, a 127.5B-total / 5.1B-active hybrid MoE (BailingMoeV3, llama.cpp arch bailingmoe3).

Every file here contains only sm_120-safe tensor types — no iq1_s, iq2_s or iq3_s, which are broken on NVIDIA Blackwell (RTX PRO 6000, RTX 5090, RTX 5070 Ti). All were quantized from the BF16 master, never from another quant, and declare a verified 256K context window.

⚠️ Re-download if you pulled these before 2026-08-17

These files were republished on 2026-08-17 and the previous ones are obsolete in two independent ways:

1. They will not load. Upstream llama.cpp merged official support (#26608) as arch bailingmoe3. The old files declared bailing-hybrid, a name we invented before support existed, and used tensor names (ssm_f, ssm_g) that did not survive review. No patch is needed any more — build stock llama.cpp.
2. They were missing the SwiGLU clamp — a real quality bug, not a rename. See The SwiGLU clamp below.

Nothing about the weights themselves changed: the fix was metadata, so the tensor data in these files is byte-for-byte what it was. You still need the new files, because the old metadata cannot be loaded or corrected in place.

Which one do I want?

Reference PPL 4.1516 ± 0.0461, from the BF16 master with the clamp active.

• Q5_K_XXL — the most accurate build that fits a 96 GB card at the full 256K window. Recommended for a dedicated RTX Pro 6000; it leaves ~1.4 GiB spare.
• Q5_K_M — the balanced pick, and the right one if anything else shares the GPU (~10.5 GiB spare).
• Q5_K_XL — in between, ~5.9 GiB spare.
• Q4_K_M — smallest. The Q4→Q5 step is about twice the Q5→Q8 step on both mean KLD and top-1, so if you are deciding where to spend memory, that is the one that matters.
• Q8_0 — reference fidelity. Needs expert offload or two GPUs.

KV cache is unusually cheap here — only 7 of 42 layers are MLA, and those are compressed — so the full 256K window costs under 2 GiB. Weights are what limit you on this model, not context.

The SwiGLU clamp

config.json carries expert_swiglu_limit_list and share_expert_swiglu_limit_list, which clamp the SwiGLU activations on the deepest layers: routed experts at 4.0 on layers 35-41, shared experts at 5.0 on 34-39 and 7.0 on 40-41. The op is silu(gate).clamp(max=L) * up.clamp(-L, L).

Files published before 2026-08-17 did not carry these values, so the clamp never fired. The mistake is worth describing, because the same trap is waiting in any port of this family: inclusionAI's own modeling_bailing_moe_v3.py does not implement it — BailingMoeV3MLP.forward is a plain SwiGLU with zero references to the limit lists — so reading the reference implementation is not enough to catch it. vLLM's bailing_moe_v3.py implements it (SwigluStepAndMul), and so does the merged llama.cpp. Two independent runtimes against one incomplete reference file.

Measured on Q5_K_XXL, clamped against unclamped, on identical weights:

The effect is rare but severe: the median barely moves while the 99.9th percentile goes up by 16×, which is exactly what a clamp on outlier activations should look like. Perplexity drops 5.4%.

At full precision the same thing shows up independently — the BF16 master scores PPL 4.1516 with the clamp against 4.3970 without it, on identical weights. A 5.6% perplexity improvement is not what a spurious operation does.

There is a second-order benefit: clamping also makes the model easier to quantize. Measured against a matched reference, every tier's divergence roughly halves versus the old unclamped-vs-unclamped numbers — Q5_K_XXL goes 0.00919 → 0.00507 mean KLD, Q8_0 0.00668 → 0.00297 — because bounding the activations removes the outlier paths where quantization error was amplified.

Running it

Point llama.cpp at shard 00001; it finds the rest. Omitting -c gives the full 262,144-token window.

./build/bin/llama-server -m Ling-3.0-flash-Q5_K_XXL-00001-of-00003.gguf -ngl 99

Reasoning is on by default — allow enough completion tokens or you will get truncated or empty content. Sampling per the upstream model card: temperature 0.6, top_p 0.95, top_k 20.

If -ngl 99 does not fit your card you have two dials, and context is the cheaper one to keep:

• Offload experts to CPU — -ncmoe N keeps the experts of N blocks in system RAM. Experts are 97% of the weights, so this is the big lever. Costs speed, not context.
• Reduce context — saves comparatively little, since the whole 256K window is under 2 GiB of KV. Reach for this last.

Rough sizing, weights + KV at full 256K: Q4_K_M ≈ 72 GiB, Q5_K_M ≈ 86 GiB, Q5_K_XL ≈ 91 GiB, Q5_K_XXL ≈ 95 GiB, Q8_0 ≈ 128 GiB, plus a couple of GiB of compute buffers.

Config for a single 96 GB card (RTX PRO 6000 Blackwell)

Specific to a 96 GB card with nothing else resident

./build/bin/llama-server \
  -m Ling-3.0-flash-Q5_K_XXL-00001-of-00003.gguf \
  -ngl 99 -c 262144 -ub 512 --no-mmap

Measured on these exact files, upstream b10470: 95,696 MiB resident at the full 256K window with no CPU offload, 125 tok/s decode. Prefill was ~2,040 tok/s on a 235K prompt when last measured, before the migration.

-ub 512 is load-bearing at this size. -ub is the compute-buffer dial and its price scales with context: at 262,144 each ubatch token costs ~0.695 MiB, since the attention mask is ctx × ubatch and is materialised even with -fa on (measured: 256→512 = +178 MiB, 512→1024 = +356 MiB). At 32K it is roughly a tenth as expensive. Lowering -ub costs prefill throughput and leaves decode untouched. When tuning this close to the limit, confirm with an actual generation rather than a successful load — CUDA allocates some workspace lazily on the first matmul.

Context: 256K, not 131K

These GGUFs give you 262,144 tokens by default. You do not need any flag.

This used to require a patch on our side: config.json originally shipped max_position_embeddings: 131072, contradicting inclusionAI's own card, which documents a training schedule of 8K → 32K → 256K and benchmarks at a 256K window. inclusionAI has since corrected the config to 262144, so these files get the right window straight from the converter.

It matters because llama.cpp caps a slot to the GGUF's context_length:

the slot context (262144) exceeds the training context of the model (131072) - capping

Verified by needle-in-a-haystack out to 247,583 tokens.

To use the old 131072 value instead:

--override-kv bailingmoe3.context_length=int:131072

Files

Each quant is split into shards for resumable download.

The bailing-hybrid-llama.cpp.patch that used to live here has been removed. It was written against 6ea215d17, before upstream support existed, and no longer applies to master. Build stock llama.cpp instead.

How these were made

Experts are 97.12% of all parameters, so the recipe spends its budget there and protects everything else cheaply: all 3.67B non-expert parameters are pinned to q8_0 for only 3.6 GiB. That covers the KDA gates (ssm_f_a/ssm_g_a), the MLA absorption tensors (attn_k_b/attn_v_b), the head-wise attention gate, embeddings and output. Norms, the router (ffn_gate_inp), exp_probs_b and the KDA scalars (A_log, dt_bias, conv1d) stay F32.

Pinning attn_k_b also avoids a silent fallback: its ncols is 128, not divisible by 256, so K-quants degrade it to q5_1/q4_1.

The expert projections are where the tiers differ:

down_proj is promoted first because it is the output side of the SwiGLU, where error propagates instead of being gated away. up_proj follows: in down(silu(gate(x)) ⊙ up(x)) an error in up passes through the product with its relative magnitude intact, while an error in gate is first attenuated by silu.

Q5_K_XXL stops where it does because of a hardware ceiling. Promoting up on all 40 MoE layers costs another 892 MiB, which forces -ub 256 and drops prefill by roughly a quarter for no measurable accuracy gain. Full Q6_K experts needs ~101.4 GiB resident against 97,887 MiB of card, and since the entire 256K KV cache is only ~2 GiB, no amount of context reduction rescues it. The seven layers left at q5_K are the seven lowest by imatrix activation energy — Σ(Act²) of 11K–24K against 193K at layer 40.

imatrix from 892K tokens of mixed code, agent traces and technical prose. With top-8 routing over 512 experts, 40 of 120 expert tensors reached only 98.24–99.80% coverage — 1–9 experts per tensor never routed to, concentrated in the deepest layers. Those tensors (blk.36/40/41) are quantized one step higher so missing importance data is paid for in bits, costing ~1.1 GiB.

The imatrix was collected before the clamp was known about, so it was re-checked rather than assumed: collecting a matched pair with and without the clamp moves per-column importance only in the extreme head (Spearman ≥ 0.9992, top-1% column overlap 97–98%, and only 4 of 573 tensors shifting more than 1% in total activation energy — all of them ffn_down_exps on layers 38–41). Bit allocation lands in the same place, so these quants were not rebuilt.

Validation

From a single RTX PRO 6000 Blackwell 96 GB. Q4/Q5 run all-GPU; Q8_0 with -ncmoe 16.

• Needle-in-haystack 12/12 at depths 10/50/90% across 6.6K, 28K, 61K and 123.4K tokens, and 12/12 again at 175K and 247.6K — which is what justifies the 262,144 context_length these files declare. Measured on Q4_K_M, Q5_K_M and Q8_0; Q5_K_XXL passes at 235,379 tokens and Q5_K_XL at 179,167.
• Coherence 6/6 — factual recall, arithmetic, the bat-and-ball trap, code generation, exact-word instruction following, translation.

Divergence figures in the table above come from llama-perplexity --kl-divergence against reference logits from the BF16 master itself, not a Q8_0 proxy — quant-vs-quant comparisons overstate damage because the errors are correlated. 160 chunks × 512 tokens of held-out wikitext-2 (~82K tokens, zero verbatim overlap with the imatrix corpus). The reference was regenerated on 2026-08-17 with the clamp active; figures published before that date were measured against an unclamped reference and do not carry over.

That ordering is solid and monotonic in bit-width. A 20-instance multi-hop reasoning probe at ~111K tokens could not separate the Q5 tiers from one another, so treat KL divergence as the evidence for quality ordering, and do not read the gaps between adjacent Q5 tiers as a promise of visibly better answers.

Not measured: tool calling, and reasoning quality between 128K and 256K — retrieval is verified to 247.6K, but reasoning was only probed at ~111K.

MTP: works, predicts well, and is still slower — leave it off

The layer-42 nextn head is present in every file here. Upstream bailingmoe3 can actually use it (--spec-type draft-mtp), which our own port never could, so it is now measurable. On Q5_K_M at 32K, single stream:

The head drafts well — 96% accepted at depth 1 — and it is a ~20% net loss anyway, at every depth. The flat curve is the tell: if the cost scaled with drafted tokens, depth 1 would sit near baseline. It does not, so this is a fixed per-step cost of running the MTP graph at all — an extra graph launch, a 512-expert gather, and a projection over a 157K vocab, paid once per decode step regardless of how many tokens it drafts.

Worth stating plainly because it is the opposite of the usual intuition: acceptance rate is not the metric. Measure tokens per second.

Enabling MTP also costs ~2 GiB of VRAM, since the layer-42 tensors go from "unused tensor — ignoring" to actually allocated. That alone rules it out alongside Q5_K_XXL on a 96 GB card, which has ~1.4 GiB spare.

Architecture notes

42 layers: 35 KDA (Kimi Delta Attention) + 7 gated MLA at layers 5/11/17/23/29/35/41, plus an MTP/nextn head at layer 42. 512 routed experts, top-8 plus 1 shared, noaux_tc grouped routing (8 groups, 4 used), sigmoid scoring.

Three things differ from Kimi-Linear and matter to anyone porting this:

• kda_safe_gate replaces -exp(A_log)*softplus(·) with lower_bound * sigmoid(exp(A_log) * (f(x) + dt_bias)), lower_bound = -5.0. A_log is stored as +exp(A_log); the sign lives in bailingmoe3.kda.gate_lower_bound.
• rope_interleave: true resolves to llama.cpp's NORM rope, not the NEOX that DeepSeek-style MLA usually uses.
• attention.g_proj exists on both layer types with different shapes — [4096,2560] KDA full-rank output gate vs [32,2560] MLA head-wise attention gate. The converter sends them to different tensors (ssm_g_a and attn_gate); a name→enum table cannot express that.

Building llama.cpp

Any master build from 2026-08-17 onward has bailingmoe3 support. No patch.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

License

MIT, following the base model.