Downloads · 30 days
118
2% of all-time downloads
Incarnas/Qwen3.5-122B-A10B-NVFP4-MTP-GGUF
Qwen3.5-122B-A10B-NVFP4-MTP-GGUF is a image-text-to-text model from Incarnas. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as apache-2.0.
NVFP4 GGUF of Qwen/Qwen3.5-122B-A10B with the MTP (Multi-Token Prediction) head retained. Structurally identical to the sibling artifact Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF plus one additional layer block carrying t…
Downloads · 30 days
118
2% of all-time downloads
All-time downloads
7.2K
Public
Repo size
82.9 GB
Likes
2
Public
Click a slice to open those files.
.gguf82.9 GB · 100%
From the Hugging Face model README
NVFP4 GGUF of Qwen/Qwen3.5-122B-A10B with the MTP (Multi-Token Prediction) head retained. Structurally identical to the sibling artifact Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF plus one additional layer block carrying the MTP draft weights, enabling self-speculative decoding via llama.cpp's --spec-type draft-mtp path.
122B total parameters, ~10B active per token via 256 MoE experts (8 routed plus 1 shared). 48 base transformer blocks plus 1 MTP block (block index 48). 77 GiB main GGUF, 871 MiB mmproj sidecar carrying the vision tower.
For reproducibility / audit / debugging details — full bug-discovery narrative, per-test bench tables, hardware specs, failure modes encountered during this run, and step-by-step reproduction recipe — see the companion TECHNICAL_REPORT.md (or the PDF version on the methodology repo).
The full methodology toolchain — the converter patch, bench scripts, raw bench JSONs, power telemetry CSVs, and gsm8k samples — lives at bit-incarnas/nvfp4-mtp-conversions (v1.0).
1867a0c69, first contained in release tag b9208). The artifact itself is unchanged; reproducers building against b9208+ no longer need the local patch step. The patch remains documented in the methodology repo for anyone reproducing against the original pre-merge b9187 build.benchmark_results/ tree on the methodology repo, where copies of the bench JSONs + power CSVs from this release's bench window are committed.b9187 (post-PR-22673 merge), which added Qwen3.5-MoE MTP tensor remap in convert_hf_to_gguf.py. Bench: AR decode 79.8 t/s, MTP n=2 decode 116.5 t/s (+46%), spec acceptance 2.46 eff tok/decode.The standard Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF GGUF is built from the same NVFP4 source weights but with MTP tensors stripped (the converter version at the time of that release did not yet handle the mtp.* prefix for the qwen3_5_moe arch). PR #22673 fixed the converter at merge.
This artifact retains the MTP block at layer 48. Concretely the additions are:
| Tensor | Shape | Purpose |
|---|---|---|
blk.48.nextn.eh_proj.weight | [6144, 3072] | Embedding+hidden projection for the MTP head |
blk.48.nextn.enorm.weight | [3072] | Embedding norm |
blk.48.nextn.hnorm.weight | [3072] | Hidden-state norm |
blk.48.nextn.shared_head_norm.weight | [3072] | Shared head norm |
blk.48.attn_*, blk.48.ffn_* (16 tensors) | various | Full attention + MoE FFN of the MTP block |
Together these add ~6 GiB on disk vs the no-MTP sibling. GGUF metadata: qwen35moe.block_count = 49, qwen35moe.nextn_predict_layers = 1.
Source weights: txn545/Qwen3.5-122B-A10B-NVFP4 (NVIDIA ModelOpt NVFP4 safetensors, ~74 GiB, 149,765 tensors including 785 under the mtp.* prefix).
Converter: llama.cpp convert_hf_to_gguf.py at commit 0253fb21f (build b9187), the first build with PR #22673 (MTP support) on master.
A one-line local patch to conversion/qwen.py (committed at patches/qwen35_mtp_bid_fix.patch in the methodology repo) was required on top of mainline. _Qwen35MtpMixin.modify_tensors() (around line 597 at commit 0253fb21f) remaps the tensor NAME from mtp.layers.{bid} to model.layers.{bid + n_layer} but doesn't update the bid argument before yielding to super. The Qwen2MoE expert-stacker (Qwen2MoeModel.modify_tensors, around line 87) uses bid as the key into its expert cache and rebuilds expected tensor names from bid directly, so the renamed-but-bid-unchanged tensor gets cached at the wrong slot and the stacker tries to look up layer-0 keys in a cache populated with layer-48 keys, crashing with KeyError: 'model.layers.0.mlp.experts.0.down_proj.weight'. The fix adds one line:
--- a/conversion/qwen.py
+++ b/conversion/qwen.py
@@ class _Qwen35MtpMixin:
def modify_tensors(self, data_torch, name, bid):
if name.startswith("mtp."):
n_layer = self.hparams["num_hidden_layers"]
if name.find("layers.") != -1:
assert bid is not None
name = name.replace(f"mtp.layers.{bid}", f"model.layers.{bid + n_layer}")
+ bid = bid + n_layer
else:
...
The bug doesn't manifest for text-only Qwen3.5-MoE sources (e.g., the ones Unsloth converted for their MTP releases) because their safetensors use the bare model.layers.{N}.* naming and the MoE expert-stacker happens to consume the layer-0 cache before the MTP path runs. It does manifest for Qwen3_5MoeForConditionalGeneration (multimodal) sources where tensors arrive under model.language_model.layers.{N}.* and the strip-prefix path changes the order in which the cache fills. Merged upstream as ggml-org/llama.cpp#23237 on 2026-05-18 (master commit 1867a0c69, first release tag b9208). Reproducers building from llama.cpp b9208+ no longer need to apply the patch -- the fix ships in mainline. The patch in patches/qwen35_mtp_bid_fix.patch remains available for anyone reproducing against the original pre-merge b9187 build that this v1.0 GGUF was produced from.
llama.cpp converter changes that apply to this artifact:
--spec-type draft-mtp runtime support.BLACKWELL_NATIVE_FP4=1 at runtime on a supported GPU.input_scale from FP8-dequanted modelopt tensors.Vision tower preserved as a separate mmproj GGUF, identical to the no-MTP sibling's mmproj (vision tower itself is unchanged between releases). Chat template embedded. Thinking mode is enabled by default per Qwen's chat template.
ModelOpt's hf_quant_config.json excludes the following modules from NVFP4. They stay at the source dtype (BF16 with FP8 scales on the quantized portions):
lm_head (output projection).mlp.shared_expert_gate modules (MoE router gates).KV cache: kv_cache_quant_algo: FP8 per the source config. NVFP4 group size: 16 elements. NVFP4 scale dtype: FP8 (E4M3).
If you try to convert your own NVFP4 source and hit:
NotImplementedError: Quant format 'nvfp4-pack-quantized' for method 'compressed-tensors' is not yet supported
the cause is that two different toolchains both produce things called "NVFP4" and only one is supported by llama.cpp today.
| Toolchain | quant_method in config | llama.cpp converter |
|---|---|---|
| NVIDIA ModelOpt | modelopt (with quant_algo: NVFP4) | Supported |
| llm-compressor / compressed-tensors | compressed-tensors (with format: nvfp4-pack-quantized) | Not yet supported |
Check quant_method in the source config.json before downloading. ModelOpt sources work today.
| File | Size | SHA256 | Contents |
|---|---|---|---|
Qwen3.5-122B-A10B-NVFP4-MTP.gguf | 77 GiB | 524c90b30067f890bb5e858b2ac868be3f06b64864eeab550a6fb415f9ef3063 | Main LM with MTP block. NVFP4 Linear weights. BF16 exclude list. MoE routing. Embedded chat template. Filetype: MOSTLY_NVFP4. 1475 tensors total. |
mmproj-Qwen3.5-122B-A10B-NVFP4.gguf | 871 MiB | 451c74191a03f3b916db44e2f364415c5fab8ff5a02d13fc5c79dde3ee0b54c8 | Vision tower with qwen3vl_merger projector. Loaded via --mmproj. Byte-identical to the no-MTP sibling's mmproj. |
MTP self-speculative decoding (single-user, max decode rate):
llama-server \
-m Qwen3.5-122B-A10B-NVFP4-MTP.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-ngl 999 \
-c 32768 \
-fa on \
-ctk q8_0 -ctv q8_0 \
--port 8080
Vision (no MTP, multimodal input enabled):
llama-server \
-m Qwen3.5-122B-A10B-NVFP4-MTP.gguf \
--mmproj mmproj-Qwen3.5-122B-A10B-NVFP4.gguf \
-ngl 999 \
-c 32768 \
--port 8080
Docker:
docker run --rm -it --gpus all \
-v $PWD:/models \
-p 8080:8080 \
-e BLACKWELL_NATIVE_FP4=1 \
ghcr.io/ggml-org/llama.cpp:server-cuda \
-m /models/Qwen3.5-122B-A10B-NVFP4-MTP.gguf \
--spec-type draft-mtp --spec-draft-n-max 2 \
-ngl 999 -c 32768 -fa on
Toggle thinking mode via enable_thinking in chat params if your client supports it.
These are PR #22673 limitations as of the merge commit:
--mmproj is incompatible with --spec-type draft-mtp. The loader hangs during init. Pick one of the two modes per server instance.n_parallel must be 1 under --spec-type draft-mtp. Multi-slot serving disables MTP.cache_reuse is auto-disabled by the server when --spec-type draft-mtp is active.--spec-draft-n-max sweet spot is 2 on this artifact at the measured envelope. n=3 and n=4 are within 1% of n=2 on decode rate but use more VRAM for the draft buffer and slightly more compute per accepted token.Measured 2026-05-16 on Blackwell Pro 96 GiB (RTX PRO 6000 Blackwell Max-Q, 300 W TDP) at llama.cpp commit 0253fb21f (build b9187), 256k context, KV cache q8_0/q8_0, --flash-attn on, BLACKWELL_NATIVE_FP4=1. Decode and prefill numbers are medians from tools/bench_decode_prefill.sh with 3 reps per metric. AR mode: --spec-type none. MTP mode: --spec-type draft-mtp --spec-draft-n-max 2. Power and clock telemetry sampled at 500 ms intervals during each bench run.
| Test | AR | MTP n=2 | Δ |
|---|---|---|---|
| Short decode (29 tok output) | 81.31 t/s | 128.01 t/s | +57.4% |
| Long decode, thinking on (2048 tok) | 79.77 t/s | 116.45 t/s | +46.0% |
| Long decode, thinking off (350-376 tok) | 80.09 t/s | 95.13 t/s | +18.8% |
The long-decode AR baseline of 79.77 t/s matches the no-MTP sibling's published 79.8 t/s within noise — the conversion is structurally additive, non-MTP path performance is preserved.
MTP wins most on the thinking-on regime (+46%) because the long structured reasoning trace is highly predictable: the draft head accepts many tokens per verifier step. Plain decode also gains (+19%) but less, because non-thinking output is more locally entropic. Short decode shows the largest absolute speedup (+57%) because draft-stage warmup costs are amortized over fewer total tokens.
| Counter | Δ over the bench run |
|---|---|
| Tokens predicted total | 7,324 |
| Decode calls | 2,978 |
| Effective tokens per decode call | 2.4594 |
Effective tokens per decode = Δtokens_predicted / Δn_decode from /metrics. AR would yield ~1.0; uniform 2-token speculation with 100% acceptance would yield 3.0 (1 verified + 2 accepted drafts). 2.46 corresponds to roughly 73% draft acceptance on this workload at --spec-draft-n-max 2. Acceptance is regime-dependent; thinking-on reasoning traces accept more than terse instruction-following.
| Test | AR | MTP n=2 |
|---|---|---|
| Short prefill TTFT (24 tok) | 51 ms | 55 ms |
| Long cold prefill rate (7,906 tok) | 4,137 t/s | 3,386 t/s |
| Long cold prefill TTFT (7,906 tok) | 1,911 ms | 2,335 ms |
| Warm cache-reuse TTFT (3-tok suffix on 7,910-tok prompt) | 27 ms | 35 ms |
MTP carries a small per-step overhead during prefill — the drafter graph runs alongside the verifier graph — so the long-prefill rate is ~18% lower under draft-mtp. Decode amortizes this many times over: for any output longer than ~30 tokens the MTP path is net faster wall-clock. For pure short-prompt-short-output workloads (e.g. classifier-style chains) AR is the better pick.
Carried over from the no-MTP sibling (the prefill compute path is structurally unchanged — MTP affects per-step compute during decode, not during the prefill pass):
| Prompt tokens | Prefill t/s |
|---|---|
| 8,000 | 4,159 |
| 32,000 | 3,498 |
| 65,000 | 2,910 |
| 130,000 | 2,303 |
| 229,000 | 1,812 |
Clean monotonic decay. Per-token prefill compute grows with KV state.
| Mode | Resident | Free (of 95 GiB usable) |
|---|---|---|
| AR | 82.4 GiB (84,330 MiB) | 12.6 GiB |
| MTP n=2 | 88.1 GiB (90,182 MiB) | 6.9 GiB |
| Δ | +5.7 GiB | -5.7 GiB |
The +5.7 GiB delta breaks down approximately as: ~4 GiB MTP-block weights (resident on top of the no-MTP sibling's 78.7 GiB) and ~1.7 GiB draft-state buffers (acceptance KV slice + drafter compute scratch, scales with context length and --spec-draft-n-max). Headroom at 256k is tight under MTP — drop to 131k context for a 4-5 GiB cushion if you also need to co-load adapters or run with --cpu-moe-draft off.
VRAM at 256k with mmproj loaded was not measured on this artifact under MTP because --mmproj and --spec-type draft-mtp are mutually exclusive at server startup. In AR mode with mmproj loaded, expect ~82.6 GiB resident (vision tower adds 248 MiB; image processing adds ~52 MiB peak per 512x512 image).
GPU telemetry sampled at 500 ms intervals during long-decode-thinking-on:
| Metric | AR | MTP n=2 |
|---|---|---|
| Samples | 202 | 152 |
| Power avg | 284.3 W | 271.9 W |
| Power median | 288.9 W | 279.8 W |
| Power p95 | 295.4 W | 287.2 W |
| Power peak | 303.3 W | 299.0 W |
| GPU clock avg | 2,203 MHz | 2,175 MHz |
| Mem clock avg | (n/a) | 13,365 MHz |
| Temp avg | 61.8 °C | 57.8 °C |
| Utilization avg | 96.0% | 91.5% |
| tok/J (long decode think) | 0.276 | 0.416 |
MTP delivers +51% tok/J on long-decode-thinking. Same wall power, more tokens per joule because the GPU spends less time in compute-saturated regime per accepted token. Lower utilization, lower temperature, lower clock under MTP also reflect that compute is no longer the binding constraint when draft acceptance amortizes it across multiple tokens per verifier step.
BLACKWELL_NATIVE_FP4=1 was set in both runs. MoE decode here is memory-bandwidth-bound rather than compute-bound, so the FP4 native tensor-core path does not materially change decode rate on this rig — the flag is still recommended on Blackwell because differences may show on workloads dominated by prefill compute (which this measurement covers but at a single prompt size).
/media/templeBLACKWELL_NATIVE_FP40253fb21f (build b9187), .devops/cuda.Dockerfile --target serverThe bench scripts used here are simple bash/python over urllib.request against the llama-server HTTP endpoint, no heavyweight harness dependencies. Three reps per metric, GPU telemetry sampled at 500 ms intervals via nvidia-smi.
Equivalent stand-alone invocations after starting the server (llama-server -m <gguf> --spec-type {none|draft-mtp} --spec-draft-n-max 2 -ngl 999 -c 262144 -fa on -ctk q8_0 -ctv q8_0 --port 8080):
# Decode + prefill + power (one run per mode)
tools/bench_decode_prefill.sh http://127.0.0.1:8080 v4-nvfp4-{ar,mtp-n2}
# Speculative-decode acceptance (wraps the above + reads /metrics deltas)
tools/spec_accept.sh http://127.0.0.1:8080 v4-nvfp4-mtp-n2
# gsm8k chat-mode probe (uses /v1/chat/completions with enable_thinking)
tools/bench_chat_math.py http://127.0.0.1:8080 v4-nvfp4-{ar,mtp-n2} --limit 5
Output JSON per run lands at reports/bench/<label>.json and reports/chat-math/<label>.{json,md}. The decode-prefill bench writes a sibling <label>.power.csv with the 500 ms telemetry sweep.
Public copies of the JSONs + power CSVs from this release's bench window are committed at benchmark_results/ in the methodology repo.
MTP is verifier-driven: each speculative token is accepted only when the base model's logits agree at that position. Output is therefore statistically equivalent between MTP-on and AR (no-spec) modes at the same sampling settings. Capability scores measured on the no-MTP sibling carry over directly to this artifact for chat-completions thinking-off, which is the regime where MTP changes wall-clock but not token identity.
A 5-sample gsm8k slice was run on this artifact under both --spec-type draft-mtp --spec-draft-n-max 2 and --spec-type none AR, using /v1/chat/completions with thinking enabled and bench_chat_math.py defaults (non-zero temperature, top-p sampling):
| # | Gold answer | MTP n=2 answer | AR answer | Outcome |
|---|---|---|---|---|
| 1 | 18 | 9 | 9 | identical wrong answer |
| 2 | 3 | 3 | 3 | identical right answer |
| 3 | 70000 | (hit 4096-tok thinking budget) | (hit 4096-tok thinking budget) | both abandoned at same point |
| 4 | 540 | 540 | 180 | divergent — sampling variance |
| 5 | 20 | 1 | 1 | identical wrong answer |
Four of five samples produced identical answers between MTP and AR. The diverging sample (Q4) reflects sampling-temperature variance at the per-token level rather than an MTP-induced quality regression; each mode samples its own trace through the same logit distribution at non-zero temperature. Sample size is too small to draw absolute-accuracy conclusions; the comparison establishes the equivalence claim, not the capability number.
Full-volume capability runs from Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF (same NVFP4 source weights, same architecture, structural delta is one additional block; the verifier-driven equivalence argument applies):
| Task | Score | n | Method |
|---|---|---|---|
| gsm8k strict (chat) | 89% | 100 | /v1/chat/completions, custom scorer |
| MATH-500 (LaTeX-aware scoring) | 88% | 100 | sympy equivalence |
| HumanEval pass@1 (chat) | 96% (48/50) | 50 | chat-mode |
| ifeval prompt-level strict (chat) | 90.0% | 100 | chat-mode |
| ifeval prompt-level loose (chat) | 91.0% | 100 | chat-mode |
| ifeval instruction-level strict (chat) | 93.3% | 100 | chat-mode |
| ifeval instruction-level loose (chat) | 94.5% | 100 | chat-mode |
| gsm8k strict (raw lm-eval) | 83% | 100 | /v1/completions, lm-eval-harness |
| HumanEval pass@1 (raw lm-eval) | 96% | 50 | raw mode |
| HumanEval+ pass@1 (raw lm-eval) | 88% | 50 | raw mode |
| MBPP+ (raw lm-eval) | 87% | 100 | raw mode |
| ifeval prompt-strict (raw lm-eval) | 44% | 200 | raw mode |
| NIAH @ 256k | 18/18 | 18 probes | 8k → 256k tiers, varied positions |
| Quality rubric (chat) | 20/20 | 20 prompts | deterministic 20-prompt sanity |
| Vision rubric | 6/6 | 6 images | geometric / OCR / gradient / multi-quadrant / counting / blank |
The chat-vs-raw gap on ifeval (90% vs 44%, +46pp) is a methodology artifact, not a model property. Raw mode bypasses the chat template's role framing, so the model never engages instruction-following machinery for that task type. The chat-mode column is the right reference for chat-client and agentic-harness use cases; the raw column is the right reference for HF-leaderboard comparability. See Methodology notes below.
--spec-draft-n-max sweet-spot rationaleThe choice of n=2 as the production setting is supported by sweeps on related artifacts at the same b9187 build:
| Artifact | n=1 | n=2 | n=3 | n=4 | n=5 | Peak |
|---|---|---|---|---|---|---|
| Qwen3.5-122B-A10B Q4_K_XL (no-MTP sibling family) | 130.85 | 152.27 | 150.25 | 151.08 | 143.72 | n=2 |
| Qwen3.6-27B dense Q5_K_XL | 79.43 | 82.61 | 82.40 | 78.07 | 74.95 | n=2 |
Both peak at n=2 with a 1-3% plateau through n=4. This NVFP4-MTP artifact was measured at n=2 only; the broader sweep was not repeated because the cross-artifact pattern is established. Long-decode-thinking is the regime where n=2 wins by the largest margin; pure short-output regimes may benefit from n=3 or n=4 at the cost of more VRAM.
--spec-type draft-mtp (statistical equivalence holds in principle and on the 5-sample spot-check; a 100-sample sweep is a planned follow-up).--spec-draft-n-max sweep beyond n=2 (peak location inherited from cross-artifact data above).Every datapoint above falls into one of three categories:
| Category | Datapoints | Source |
|---|---|---|
| Measured fresh on this artifact (2026-05-16) | Decode (short / long think / long plain) for AR + MTP n=2; long-cold-prefill rate; short-prefill TTFT; warm-cache-reuse TTFT; spec acceptance; VRAM at 256k AR + MTP; power / GPU clock / temperature / utilization at 500 ms granularity; 5-sample gsm8k MTP-vs-AR equivalence | This rig, this GGUF, build b9187 |
| Inherited from the no-MTP sibling | Prefill curve 8k → 229k tokens; full-volume gsm8k / MATH-500 / HumanEval / ifeval / MBPP+ (chat and raw); NIAH @ 256k; quality rubric; vision rubric | Sibling Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF at convert-script commit 1ec7ba0c1; same source weights, same architecture except for the additional MTP block. Verifier-driven equivalence justifies the inheritance for capability scores. |
| Cross-referenced from sibling artifacts | --spec-draft-n-max sweep peak at n=2 | Qwen3.5-122B-A10B Q4_K_XL canonical and Qwen3.6-27B dense Q5_K_XL, both at build b9187, sweep n=1..5 with the same bench_decode_prefill.sh + spec_accept.sh pattern. Both peak at n=2; this artifact was therefore measured at n=2 only. |
Raw bench JSONs from this run live in the publish-pipeline working tree (see TECHNICAL_REPORT.md §8 for absolute paths), not in this HF repo. The companion TECHNICAL_REPORT.md, linked at the top of this card, carries the full bug-discovery narrative, per-test bench tables, hardware specs, failure modes encountered during this run, and the step-by-step reproduction recipe.
This release ships under the bit-incarnas/nvfp4-mtp-conversions methodology repo (v1.0), which carries the full technical report (markdown | PDF), the local converter patch, raw bench JSONs + power telemetry, and the reproduction recipe.
The chat-vs-raw methodology gap and its implications for ifeval / instruction-following tasks are the same as on the no-MTP sibling. See Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF#methodology-notes for the full breakdown, raw-mode tables, and the bit-incarnas/chat-vs-raw-methodology reproduction recipe.
This GGUF was produced by llama.cpp commit 0253fb21f (build b9187). It requires a llama.cpp build with PR #22673 merged (Qwen3.5-MoE MTP runtime support). Builds older than 2026-05-16 will load the model but cannot activate --spec-type draft-mtp; in that case the file falls back to behaving exactly like the no-MTP sibling and the MTP-block weights go unused.
For reproducing the conversion (not for running this GGUF): the v1.0 conversion required an additional one-line converter patch on top of the b9187 build. As of 2026-05-18 that patch has been merged upstream as PR #23237 in master commit 1867a0c69; the first release tag containing the fix is b9208. Reproducers using b9208+ can skip the local patch step entirely.
convert_hf_to_gguf.py (with a one-line local fix to _Qwen35MtpMixin.modify_tensors() for multimodal-prefix sources)