Ling-3.0-flash-VL-MTP-ROCmFP4-GGUF

Ling-3.0-flash-VL + MTP — ROCmFP4 for AMD Strix Halo (gfx1151)

inclusionAI/Ling-3.0-flash-VL (124B total / 5.5B active, native multimodal) quantized to ROCmFP4 for AMD Strix Halo, with a multi-token-prediction head grafted in from its text sibling — and with the server fix that lets MTP and images work at the same time, which they do not in stock llama.cpp.

Everything below was measured on the shipped files on one Ryzen AI Max+ 395 (Radeon 8060S, 128 GiB unified). Numbers come from the built GGUFs and from server logs, never from estimates.

Which file should I use?

Take STRIX_LEAN. It has the best perplexity and the best prefill, at 4 GB less than COHERENT.

If your runtime cannot do draft-mtp, take the no-MTP file instead — see Running without MTP.

⚠️ COHERENT is the worst of the three on this model even though it is the largest and the highest bit-width (4.70 bpw vs 4.38). Its PPL is 0.80 worse than STRIX_LEAN — roughly 3.5σ against the ±0.23 error bars — and it is slower on both prefill and decode. It ships for completeness; it is not the "quality" option here. Tier ranking is model-dependent and has to be measured, not assumed.

FAST is a fair pick if you only care about decode: +6% TG over STRIX_LEAN on Vulkan0 at 0.2 GB less, for 0.21 PPL.

PPL = wikitext-2 wiki.test.raw, 20 chunks, -c 2048 -b 2048, ROCm0, identical settings per tier. PP/TG = prefill / decode tokens/s on a 659-token prompt generating 128 tokens, MTP at n-max 1, quiet box (nothing else running — a single busy CPU thread measurably moves these numbers).

Quick start

# ROCm0 — best prefill: long prompts, agentic/tool work
llama-server -m Ling-3.0-flash-VL-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-Ling-3.0-flash-VL-BF16.gguf \
  -dev ROCm0 -ngl 999 -fa on --jinja --parallel 1 \
  -c 8192 -b 2048 -ub 1024 \
  --spec-type draft-mtp --spec-draft-n-max 1 --spec-draft-n-min 0 --spec-draft-p-min 0.0

Swap -dev ROCm0 for -dev Vulkan0 if your traffic is chat / long generations (see Speed).

Speed: making it fast

--spec-draft-n-max 1, not the default 16. Measured on STRIX_LEAN, ROCm0, same file and flags, 337-token prompt, 160 tokens generated, quiet box:

Longer drafts lose. Acceptance collapses past the first position (0.24 at position 2, 0.094 at position 3) while draft generation cost keeps climbing (615 → 662 → 855 ms over the run), so n-max 2 and 3 land below running without MTP at all.

MTP costs prefill: 576.7 → 534.3 tok/s (−7.4%) on the same measurement, because the draft head runs over the prompt too. Net effect favours chat and agentic turns over long-prompt batch work.

Backend: ROCm0 prefills, Vulkan0 decodes

Both backends, all three tiers, same prompt (659 tokens → 128 generated), MTP n-max 1:

ROCm0 wins prefill by 25–35%; Vulkan0 wins decode by 9–11%. Pick per workload.

The grafted MTP head

Ling-3.0-flash-VL ships without a multi-token-prediction head. Its text sibling inclusionAI/Ling-3.0-flash carries one as block 42, and every text dimension matches between the two checkpoints — vocab 157184, hidden 2560, 42 layers, 512 experts, kv_lora_rank 512, qk_nope_head_dim 128, qk_rope_head_dim 64, v_head_dim 128, 32 attention heads, rope_theta 6e6, layer_group_size 6, first_k_dense_replace 2. The only difference is num_nextn_predict_layers, which is the head.

So the head transplants exactly: 1,553 donor tensors (5.72 GiB BF16) added as one extra shard on a hard-linked copy of the VL checkpoint, num_nextn_predict_layers: 1 set in text_config, then converted. The result is 938 tensors / 43 blocks / nextn = 1 against 917 / 42 without the graft, and every shipped file was verified to still contain blk.42.* after quantization.

Vision — and MTP at the same time

In stock llama.cpp, loading an MTP head and --mmproj together crashes the server on the first image. The server processes the image into the draft context as well as the target, the draft context has no vision tower, and the failure path is a hard GGML_ABORT. On a sibling model this shows up as missing MTP boundary for seq_id=0 pos=17 → failed to process speculative batch; the MTP state is never advanced across an image's positions, which jump because the vision path uses M-RoPE.

These files are built to be served by a fixed server that advances the drafter across image positions using the model's own image_token_id (157157) plus the target's pre-norm hidden states, replicating the M-RoPE 2D positions exactly, and degrades to no-draft instead of aborting if that is ever impossible. Measured with the head loaded and --mmproj attached:

• a 789-token image request answered correctly in 7.2 s, server alive, no abort in the log
• MTP still drafting after the image — acceptance went 0.281 → 0.424 (per-position 0.909 / 0.364), so the draft KV stayed aligned rather than being silently poisoned

⚠️ This needs the patched server. On an unpatched build, serve images without --spec-type. The patch is a server change, not a weights change — the GGUFs here are fine either way.

Vision example output from …-STRIX_LEAN.gguf (real image, enable_thinking: false):

"The image displays the logo for LLaMA C++, featuring the white text 'LLaMA' followed by an orange stylized 'C++' symbol against a black background."

Running without MTP

Ling-3.0-flash-VL-Q4_0-ROCmFP4-STRIX_LEAN.gguf is the same STRIX_LEAN recipe applied to the ungrafted VL checkpoint: 917 tensors / 42 blocks, no nextn key, against 938 / 43 for the grafted files. Same protected head (output.weight Q6_K, token_embd.weight Q5_K), same vision behaviour, same 131072 context, 1.5 GB smaller.

Take it when:

• your runtime has no draft-mtp support, or
• you are on an unpatched server and want images — no MTP head means no MTP+image abort path, or
• you want the smaller resident footprint and do not need speculation.

llama-server -m Ling-3.0-flash-VL-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-Ling-3.0-flash-VL-BF16.gguf \
  -dev ROCm0 -ngl 999 -fa on --jinja --parallel 1 -c 8192 -b 2048 -ub 1024

⚠️ Its speed is left blank in the table above on purpose. It has not been re-measured under that table's protocol (659-token prompt, 128 generated, quiet box), and quoting a number taken under different settings would make the tiers look comparable when they are not. The closest honest figure is the off row in the MTP table — 33.29 tok/s decode — measured on the grafted file with the head disabled, which is the same computation this file performs but not the same file.

Reasoning controls

The chat template switches on enable_thinking (not reasoning_effort) and emits real <think> tags, so llama.cpp extracts reasoning_content correctly and no template shim is needed.

⛔ Thinking is ON by default. With a small max_tokens the whole budget is consumed inside <think> and content comes back empty. Either give it room or switch thinking off:

# empty content at 16 tokens with thinking on; both of these return "Paris"
curl … -d '{"messages":[…],"max_tokens":512}'                                     # thinking on
curl … -d '{"messages":[…],"max_tokens":32,"chat_template_kwargs":{"enable_thinking":false}}'

Tool calling

3/3 clean calls (get_weather {"city":"Paris"}) with --jinja, no shim, no argument repair.

Memory

STRIX_LEAN occupies ~64 GiB of GTT at -c 8192 and loads healthy in ~35 s. On a 128 GiB Strix Halo this leaves room for little else — plan it as the resident model, or run it on demand.

Quantization methodology

# 0. graft the text sibling's MTP head (block 42) onto a hard-linked copy of the VL checkpoint
#    (1,553 tensors, 6,144,582,792 bytes as one extra shard; num_nextn_predict_layers=1 in text_config)
python3 graft_mtp.py            # hf/ untouched, graft/ = hard links + the new shard

# 1. convert: MTP BF16 (938 tensors / 43 blocks) and the vision projector
python convert_hf_to_gguf.py graft --outtype bf16 --outfile Ling-3.0-flash-VL-MTP-BF16.gguf
python convert_hf_to_gguf.py graft --outtype bf16 --mmproj --outfile mmproj-Ling-3.0-flash-VL-BF16.gguf

# 2. quantize from BF16 only; read every head back by exact tensor name afterwards
llama-quantize --output-tensor-type q6_K                             …-MTP-BF16.gguf OUT Q4_0_ROCMFP4_STRIX_LEAN 8
llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K  …-MTP-BF16.gguf OUT Q4_0_ROCMFP4_COHERENT 8
llama-quantize                                                       …-MTP-BF16.gguf OUT Q4_0_ROCMFP4_FAST 8

# 3. quality: perplexity per tier on the shipped files (identical settings)
llama-perplexity -m OUT -f wikitext-2-raw/wiki.test.raw -c 2048 -b 2048 --chunks 20 -ngl 999 -fa on -dev ROCm0

⚠️ KLD against BF16 logits was NOT measured, and here is the honest reason. This fork computes BF16 MoE logits incorrectly on ROCm, so a BF16 reference must run on pure CPU — and this BF16 is 255,091,086,048 bytes against 125 GiB of RAM, which measured 187 s per chunk (3-chunk probe), i.e. about 5 hours for 100 chunks and ~8 hours for the 40×2048 reference the sibling cards use. Perplexity on the shipped files is reported instead; it ranks the tiers on the same corpus and settings.

Receipts read back from the built files (exact tensor names, never a substring match):

Where the tiers differ in the body (read from the shipped files):

Model facts (read from the GGUF headers)

general.architecture bailingmoe3 · context 131072 (the upstream README says 256K; the file says 131072) · n_embd 2560 · 43 blocks (42 + MTP) · 512 experts, 8 used, 1 shared, groups 8 of 4 · expert FFN 768 · expert_weights_scale 2.5 · leading_dense_block_count 2 · KDA head_dim 128, safe_gate true, gate_lower_bound −5.0 · rope base 6e6 · M-RoPE sections [8, 12, 12, 0] · vocab 157184 · per-layer SwiGLU clamps kept (42 entries, 7 non-zero from layer 35, value 4.0 — identical pattern to the published text model).

mmproj: 334 tensors, clip.projector_type qwen3vl_merger, vision depth 27, hidden 1152, FFN 4304, patch 16, spatial_merge_size 2 — matching the upstream vision_config.

Reproduction

Converter and runtime: our fork of ROCmFPX with the Bailing-V3 / V3-VL converter classes and the MTP-image server fix. Built with:

cmake -S ROCmFPX -B build -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build -j16 --target llama-quantize llama-server llama-imatrix llama-perplexity llama-mtmd-cli

-DLLAMA_BUILD_WEBUI=OFF is required on a box without npm — the server target otherwise fails provisioning its web assets.

Files

Images need the mmproj file alongside whichever tier you pick.

Known issues and limits

• MTP + images needs the patched server (above). Unpatched builds abort on the first image with a head loaded; serve without --spec-type there.
• Thinking on by default → empty content at small max_tokens.
• --spec-draft-n-max above 1 makes decode slower, not faster, on this model.
• COHERENT is worse than STRIX_LEAN on perplexity and on both speed axes despite being larger.
• The prompt cache is dropped around speculative boundaries (spec-boundary-mismatch, target-draft-restore-rejected in the log). Correctness is unaffected; long multi-turn sessions re-prefill more often than they would without MTP.
• Tokenizer warning at load: special_eos_id is not in special_eog_ids — inherited from the upstream tokenizer config; generation stops correctly in every test here.
• The no-MTP file has no published perplexity or speed figures yet; only its header receipts are verified. Everything else about it matches the grafted STRIX_LEAN build.
• No imatrix variant is published for this model.

License and attribution

MIT, inherited from inclusionAI/Ling-3.0-flash-VL. The MTP head is grafted from inclusionAI/Ling-3.0-flash, also MIT. Quantized and measured on AMD Strix Halo (gfx1151).