Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

<div align="center">

Qwen3.5 122B-A10B Heretic · ROCmFP4 iMatrix

A compact, importance-calibrated ROCmFP4 build for high-memory AMD systems

**122B total · 10B active · 60.70 GiB · 28.45 tok/s native · 33.68 tok/s MTP · BF16 KLD 0.100716 · 28.11 Decode t/s ** Decode Speed 34.16% Faster Size 13.47gb Smaller.

</div>

[!IMPORTANT] This GGUF uses custom ROCmFP4 tensor types. It requires ROCmFPX or a runtime with equivalent support; stock llama.cpp, Ollama, LM Studio, and similar stock runtimes cannot load it.

I also suggest usign the template from https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates Froggeric is tha dude.

Downloads

Browse every repository file →

What it is

This is an iMatrix-calibrated quantization of the full-weight Qwen3.5 122B-A10B Heretic checkpoint. It was built with ROCmFPX's compact Q4_0_ROCMFP4_STRIX_LEAN recipe, but the hardware-oriented preset name has been left out of the public filename.

Files

The main model is complete and runs independently. The 23-tensor, one-layer MTP file is an optional external companion; no MTP layer is embedded in the main GGUF. Runtimes call this role a draft model, but Draft is omitted from the public filename because it does not mean unfinished.

Quality

The calibrated model and its uncalibrated control were replayed against the same saved distributions from the exact BF16 parent on ROCm0.

iMatrix calibration reduced mean KLD by 14.01%, reduced the PPL ratio by 0.029034, and improved same-top agreement by 0.855 percentage points. The two-sigma KLD upper bound is 0.107530, clearing the selected 0.126 performance-oriented target.

The paired replay covers 11 complete 512-token chunks and 2,805 evaluated next-token distributions. These were recovered at an exact probability-block boundary from an interrupted 32-chunk BF16 run. This is useful preliminary evidence, not a completed 32-chunk acceptance gate.

Performance

Measured on a 128 GB AMD Strix Halo system with Vulkan/RADV, 131,072 context, parallel 1, batch/ubatch 2048/1024, Q8_0 KV cache, flash attention, and MTP off. Values are medians from repeated runs.

Relative to the otherwise matched uncalibrated build, decode improved by 2.43%, 5.23%, and 2.84% respectively. Long-prompt prefill changed by -1.03%. These are single-system measurements, not general performance guarantees.

Optional MTP acceleration

The external MTP companion was tested against this exact iMatrix trunk in a matched MTP-off/on A/B. The model, runtime, context, cache, batch geometry, prompts, and companion identity were held constant; only MTP execution changed.

MTP reduced prompt-processing throughput by about 7% because the companion must catch up on the prompt. It therefore helps most when generation is long enough to repay that fixed cost. The exact identity probe was unchanged, and both the MTP-off and MTP-on lanes passed technical prose, executable code semantics, strict JSON, tool-calling, balanced-analysis, and defensive-safety smokes.

Quantization

The model was quantized once from the validated BF16 GGUF using a 339-chunk code/technical importance matrix and the Q4_0_ROCMFP4_STRIX_LEAN preset.

In this preset, attention K/V tensors retain dual-scale ROCmFP4 protection, most transformer weights use the compact FAST layout, and token embeddings/output use Q5_K. The artifact contains 879 tensors across 48 blocks and no embedded MTP layer.

Run

llama-server \
  --model Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf \
  --host 127.0.0.1 --port 8080 \
  -dev Vulkan0 --n-gpu-layers 999 \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on --batch-size 2048 --ubatch-size 1024 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja --reasoning-format deepseek

Adjust context and cache settings for your memory budget. The tested runtime was charlie12345/ROCmFPX commit a6a93765f7ce9779c13f9881164a65f7a9f31198, built in Release mode for gfx1151 with Vulkan and HIP enabled, plus a local Qwen3.5 MoE graph correction. Results with newer ROCmFPX revisions have not yet been established.

To enable the optional companion, append:

  --spec-type draft-mtp \
  --spec-draft-model Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf \
  --spec-draft-device Vulkan0 --spec-draft-ngl 999 \
  --spec-draft-type-k f16 --spec-draft-type-v f16 \
  --spec-draft-n-max 2 --spec-draft-p-min 0.6 \
  --spec-draft-p-split 0.10 --spec-draft-backend-sampling

Tested system

No vision projector is included; treat this release as text-only.

Lineage and credits

This repository exists because several people published substantial work openly. Their roles are kept distinct so downstream users can find the original sources.

• Official base model: Qwen/Qwen3.5-122B-A10B, Apache 2.0.
• Full-weight Heretic source: trohrbaugh/Qwen3.5-122B-A10B-heretic, accessed through the pinned CCSSNE mirror at fd6d6bc89e9be47577736c952dd4d1d7c830f09f.
• MTP tensor source: OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4 at 07b7c2101c33f23a84fd3c618d00f17ea12269c3; its model card reports that the MTP tensors were restored from the official Qwen checkpoint.
• Heretic software and method: p-e-w/heretic, created by Philipp Emanuel Weidmann. The source checkpoint reports Heretic v1.2.0.
• Related full-weight Heretic reference: scrunter/Qwen3-VL-235B-A22B-Thinking-heretic. This related 235B model is not the tensor source for this release.
• ROCmFP4 implementation, quantizer, and runtime: charlie12345/ROCmFPX.
• Strix Halo deployment stack: hec-ovi/llama-vulkan-strix.
• Release inspiration: plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF.
• Conversion, validation, benchmarking, and packaging: vmlinux.

Please preserve this lineage, the Apache 2.0 license, and a description of your changes when redistributing derivatives.

License

The distributed model derivative is provided under the upstream Apache License 2.0. The included LICENSE is copied from the pinned full-weight source. Runtime and tooling repositories retain their own licenses: Heretic software is AGPL-3.0, while ROCmFPX and the deployment tooling are separately licensed. No runtime source code is bundled in this model repository.