Ling-3.0-flash-GGUF

Ling 3.0 flash: GGUF

Quantizations of inclusionAI/Ling-3.0-flash: 124B total, 5.1B active, hybrid linear attention (35 KDA blocks interleaved 5:1 with 7 gated MLA blocks) over a 512-expert MoE.

Bits are placed by hand rather than by the default rules, and the controls that prove it is worth something are published next to the files. At the same size, our layout sits 31 to 41 % closer to BF16 than what llama-quantize produces on its own

These files need a TurboQuant build. bailingmoe3 is in upstream llama.cpp, but we have some important bugfixes to it. Nothing has to be compiled, see Run it.

Pick a file

Weights and context share your memory, so leave headroom below the number in the first column. Every rung of the ladder is in the full table.

Files without the AD- prefix are controls, published so the claim above can be checked. They are not meant to be used: *_STOCK is what llama.cpp picks by itself, *_FLAT is our bit budget with the differentiation switched off.

Run it

Grab the archive for your machine from release b10269-1.5.1 or newer.

wget https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant/releases/download/b10269-1.5.0/llama-turboquant-linux-x64-cuda-13.3.tar.gz
tar xzf llama-turboquant-linux-x64-cuda-13.3.tar.gz && cd llama-turboquant-*
 
./llama-cli -m AD-Q5_K_M/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --jinja -ngl 99 -c 32768

The chat template ships inside the GGUF, thinking mode and tool calling included. Sampling recommended by the authors: temperature 0.6, top_p 0.95, top_k 20.

Intel GPUs are the one gap: there is no SYCL archive yet, that path still needs a source build.

What AD means

Atomic Dynamic: the bits are placed deliberately, along three axes.

• by tensor role. The router (ffn_gate_inp) and the expert bias stay F32, because an error there changes which expert runs instead of degrading its output. Attention, the KDA gates and the shared expert stay Q8_0. output stays F16, it feeds the logits directly.
• by projection. Inside the experts, down_proj gets more bits than gate/up, it is the more sensitive half of the SwiGLU.
• by depth. The edge MoE blocks (2, 3, 39, 40, 41) get more bits than the middle ones. Routed experts are 97.1 % of the weights, so that is the only thing actually squeezed. Everything else stays high precision and costs about 4 GB in total, which is cheap insurance.

What it is worth, measured

Those rows split the win. Most of it comes from refusing to quantize the 3 % of the weights that are not experts. The per-projection and per-depth differentiation inside the experts adds the remaining 4.3 % on top.

NVFP4

Two builds, both 72.3 GB, ->safetensors for vLLM here<- and ->GGUF here<-.

Same format and same block layout in both. The AD build differs in one thing: the scale of each block is chosen by sweeping the neighbouring UE4M3 codes, laying the weights on the E2M1 grid for each candidate and scoring the error weighted by the importance matrix, with the same convention the k-quants use.

Worth knowing before you download: at this size a k-quant rung is much closer to BF16 (AD-Q4_K_S, 74.2 GB, KL 0.0318). NVFP4 buys native FP4 tensor cores on Blackwell, not accuracy.

Measurements

All numbers are measured against the BF16 baseline on held-out text that never entered the calibration corpus, on identical hardware (4x RTX PRO 6000 Blackwell). Raw logs and json: AtomicChat/Ling-3.0-flash-GGUF-metrics.

• mean KL is how far the quantized model's next-token distribution sits from BF16, averaged over tokens. Lower is better, 0 means identical.
• 99 % KL is the worst one percent of tokens. This is where a quant actually breaks.
• top-1 is how often the quant's most likely token is the same as the BF16 one.

Two pairs sit at the same size on purpose. At 79 GB, AD-Q4_K_M is better in the tail and AD-IQ4_NL in the mean. At 74 GB, AD-Q4_K_S is better in the mean and smaller, AD-IQ4_XS better in the tail and in top-1. Pick by the metric you care about.

Sizes are GB, 10^9 bytes. llama.cpp prints GiB, so AD-Q5_K_M shows up there as 83.3 GiB.

Rung names follow the community convention, not the upstream preset list: IQ4_XXS, Q5_K_L and Q4_K_L are our mixes and you will not find them in llama-quantize.

How these were built

The base is a bit-exact BF16 conversion: 877 of 917 tensors are byte-identical to the safetensors checkpoint, the remaining 40 are the MoE routers, stored as F32 instead of BF16, which is a lossless widening (max absolute difference 0.0).

The new architecture was checked layer by layer against the HuggingFace reference before any quant was produced. Over a fixed 32-token forward, the cosine similarity of the first block output is 0.99999 and the mean KL over the vocabulary is 4.8e-4, which is the noise floor between the reference GPU kernels and the llama.cpp CPU path.

The importance matrix was collected on the BF16 model, not on a quantized proxy, over 522 chunks of 4096 tokens from AtomicChat/calib-corpora.

Which tensor gets what:

Speed

4x RTX PRO 6000 Blackwell (96 GB each), full offload, llama-bench:

Consumer cards and Apple silicon will be added as those runs happen. Comparing across different GPUs is not meaningful, so every figure says which machine it came from.

On a Mac

GGUF runs natively on Apple silicon through Metal, MLX is not required:

./llama-cli -m AD-Q5_K_M/Ling-3.0-flash-AD-Q5_K_M-00001-of-00002.gguf --jinja -ngl 99 -c 32768

A 128 GB Mac Studio fits AD-Q5_K_M (89 GB) comfortably. AD-Q6_K (107 GB) needs the wired memory limit raised and leaves little room for context.

Known limitations

• MTP / speculative decoding is not wired up. The checkpoint carries one multi-token-prediction block, the converter drops it.
• NVFP4 needs Blackwell to be fast. It loads and runs elsewhere through the dequantization path, but the native FP4 tensor cores only exist on sm_100 and sm_120.
• Intel GPUs need a source build. No SYCL archive in the release yet.