Downloads · 30 days
320
9% of all-time downloads
dxx117/Qwen3.5-397B-A17B-heretic-GGUF-v2-MTP
Qwen3.5-397B-A17B-heretic-GGUF-v2-MTP is a machine learning model from dxx117. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for llama.cpp. The card lists the license as apache-2.0.
This repo contains my v2 GGUF quants of Qwen3.5-397B-A17B Heretic with MTP support included as a standard feature.
Downloads · 30 days
320
9% of all-time downloads
All-time downloads
3.4K
Public
Repo size
1.3 TB
Likes
1
Public
Click a slice to open those files.
.gguf1.7 TB · 100%
From the Hugging Face model README
This repo contains my v2 GGUF quants of Qwen3.5-397B-A17B Heretic with MTP support included as a standard feature.
These files are intended for recent llama.cpp builds with Qwen3.5 MTP support.
These quants were made from safetensors downloaded from:
RadicalNotionAI/Qwen3.5-397B-A17B-hereticThat repo is the current Hugging Face home of the Heretic model previously associated with trohrbaugh.
The quantization source used here was a local BF16 GGUF conversion of that safetensors release.
In my testing, MTP gives roughly 20-30% faster TG at the cost of about 10% slower PP. In practice, that means it is usually faster for shorter prompts, but very long prompts (for example, around 60k tokens) can still take longer overall in wall-clock time.
Heretic does not appear to retain the full native MTP tensor set in a directly usable form, so the MTP block in this repo is grafted rather than preserved natively from the Heretic release.
In practice, these files use:
Q8_0 MTP tensors or a smaller mixed-precision MTP variant (MTPmix)The grafting workflow used here is based on Buzz’s GGUF MTP transplant script:
This repo includes:
IQ3_XXS_MTPMIXIQ3_XXS_MTPQ8_0IQ4_XS_MTPMIXIQ4_XS_MTPQ8_0Q3_K_MTPMIXQ4_K_M_MTPMIXQ4_K_M_MTPQ8_0I have not been able to get Q2 quants passing my basic tests so far, so they are not included here.
These are not pure stock single-format quants.
Examples:
IQ4_XS_MTPMIX = IQ4_XS Heretic trunk + mixed-precision grafted MTP blockQ4_K_M_MTPQ8_0 = Q4_K_M Heretic trunk + Q8_0 grafted MTP blockThe suffix after the trunk name matters.
PPL results are expected within the next 48 hours.
| Quant | PPL |
|---|---|
IQ3_XXS_MTPMIX | 4.2404 |
IQ3_XXS_MTPQ8_0 | 4.2400 |
IQ4_XS_MTPMIX | TBA |
IQ4_XS_MTPQ8_0 | 4.0953 |
Q3_K_MTPMIX | 4.0900 |
Q4_K_M_MTPMIX | TBA |
Q4_K_M_MTPQ8_0 | 4.0862 |
KLD results are coming soon.
TBAMTPQ8_0 variants preserve the draft block more faithfully, but they are larger and can reduce or erase net speedup if VRAM is tight.MTPmix variants are intended to recover more practical throughput by shrinking the draft block.These files are mainly intended for:
llama.cpp-compatible local inference