Downloads · 30 days
8.1K
14% of all-time downloads
cygnal/Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF
Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF is a image-text-to-text model from cygnal. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as apache-2.0.
Quantizations of trohrbaugh/Qwen3.8-27B-heretic-ara with MTP (Multi-Token Prediction) tensors preserved (866 tensors) and vision projector (mmproj) for multimodal image understanding.
Downloads · 30 days
8.1K
14% of all-time downloads
All-time downloads
57.1K
Public
Repo size
77.3 GB
Likes
12
Public
Click a slice to open those files.
.gguf77.3 GB · 100%
From the Hugging Face model README
Quantizations of trohrbaugh/Qwen3.8-27B-heretic-ara with MTP (Multi-Token Prediction) tensors preserved (866 tensors) and vision projector (mmproj) for multimodal image understanding.
Includes ROCmFPX quants for the ROCmFPX fork of llama.cpp — ~42 t/s average decode on AMD Strix Halo (50–75% faster than stock Vulkan).
| File | Size | BPW | Format | Description |
|---|---|---|---|---|
Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf | 14 GB | 4.26 | Q4_0_ROCMFP4_FAST | Fastest — ROCmFPX Vulkan, ~42 t/s decode with MTP |
Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf | 16 GB | 4.83 | Q4_K_M | Stock llama.cpp, ~27–28 t/s decode with MTP |
Qwen3.8-27B-heretic-ara-Q6_K-MTP.gguf | 21 GB | 6.56 | Q6_K | Higher quality, stock llama.cpp |
Qwen3.8-27B-heretic-ara-ROCmFP6.gguf | 21 GB | 6.50 | Q6_0_ROCMFPX | Higher quality, ROCmFPX Vulkan |
mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf | 931 MB | — | BF16 | Vision encoder/projector (shared by all quants) |
-dev Vulkan0. ~42 t/s average decode. Best choice if you have Strix Halo and want maximum speed.trohrbaugh/Qwen3.8-27B-heretic-ara — Arbitrary-Rank Ablation (ARA) uncensoring of Qwen3.8-27B using heretic v1.2.0+custom. Parameters: start_layer 26, end_layer 56, preserve_good_behavior_weight 0.9432, steer_bad_behavior_weight 0.0009. KL divergence 0.0535, 0/100 refusals.
llama-server \
-m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
-ngl 99 -ctk f16 -ctv f16 \
-c 32768 \
--spec-type draft-mtp \
--spec-draft-n-max 3
Swap in Q6_K-MTP.gguf for higher quality. --spec-draft-n-max 3 is optimal on AMD Strix Halo (Vulkan). Try 4 on other hardware.
llama-server \
-m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
--mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
-ngl 99 -c 8192 -fa on \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--jinja
Then send images via the OpenAI-compatible API:
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
{"type": "text", "text": "Describe this image."}
]
}],
"max_tokens": 300
}'
Note: mmproj has zero impact on text-only request performance — the vision encoder only activates for image-bearing requests. Safe to load mmproj permanently.
Requires ROCmFPX fork built for your GPU.
llama-server \
-m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
-ngl 99 -dev Vulkan0 \
-c 32768 \
-ctk q8_0 -ctv turbo4 \
-fa on \
--spec-type draft-mtp \
--spec-draft-n-max 6
Critical flags:
-dev Vulkan0 — forces RADV cooperative matrix path. Without it, ROCm backend runs at ~18 t/s instead of ~42.-ctk q8_0 -ctv turbo4 — TurboQuant KV cache. Speed-neutral, dramatically reduces KV memory (enables 262K context in 20 GB vs 61 GB with f16 KV).--spec-draft-n-max 6 — optimal for ROCmFPX (vs 3 for stock Vulkan).llama-server \
-m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
--mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
-ngl 99 -dev Vulkan0 \
-c 8192 -np 1 \
-ctk q8_0 -ctv turbo4 \
-fa on \
--jinja
Important: MTP speculative decoding must be disabled for vision requests on ROCmFPX (b213) — the non-consecutive token positions from image embeddings crash MTP boundary tracking. Omit --spec-type draft-mtp when using --mmproj. Vision decode runs at ~13.6 t/s without MTP. Text-only requests will not benefit from MTP in this configuration; use a separate text-only server with MTP for maximum text speed.
llama-server \
-m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
-ngl 99 -c 32768
| Task | ROCmFP4_FAST Decode | ROCmFP4_FAST Prefill | ROCmFP6 Decode | ROCmFP6 Prefill |
|---|---|---|---|---|
| Code gen #1 | 34.5 t/s | 42.0 | 25.7 t/s | 29.5 |
| Code gen #2 | 41.6 t/s | 61.5 | 29.3 t/s | 29.7 |
| Code gen #3 | 49.5 t/s | 61.5 | — | — |
| Math | 37.5 t/s | 97.4 | 22.9 t/s | 50.4 |
| JSON extraction | 44.7 t/s | 137.1 | — | — |
| Technical | 43.8 t/s | 59.6 | 18.1 t/s | 21.4 |
| Average | ~42 t/s | ~24 t/s* |
*ROCmFP6 benchmarked under swap pressure (21 GB model at 8K ctx). Would likely hit ~30+ t/s without swap contention.
| Quant | Prefill (t/s) | Decode (t/s) | BPW |
|---|---|---|---|
| Q4_K_M-MTP | 168–172 | 27–28 | 4.83 |
| Q6_K-MTP | ~65 | ~17–23 | 6.56 |
| Q4_K_M-MTP, no MTP | 200 | 12.2 | 4.83 |
| Config | Prompt tokens | Prefill (t/s) | Decode (t/s) | MTP acceptance |
|---|---|---|---|---|
| Vision + MTP n-max 3 (simple image) | 79 | 23.7 | 13.5 | 62.6% |
| Vision + MTP n-max 3 (complex image) | 830 | 93.4 | 12.7 | 54.6% |
| Vision, no MTP | 79 | 44.9 | 12.1 | — |
Vision decode is ~13 t/s (vs 28 text-only) due to vision encoder overhead. Text-only requests are not affected by having mmproj loaded.
ROCmFPX vision (without MTP): ~13.6 t/s decode, ~85.9 t/s prefill. MTP must be disabled for vision on ROCmFPX b213.
Stock Vulkan benchmarked on llama.cpp build 0ef6e55ed (b10254). ROCmFPX benchmarked on b213/b2f5829.
| Config | Avg Decode (MTP) | Avg Prefill | File Size | Memory fit (122 GB) |
|---|---|---|---|---|
| ROCmFPX FP4_FAST | ~42 t/s | ~77 t/s | 14 GB | ✅ Comfortable |
| Stock Q4_K_M | ~27–28 t/s | ~170 t/s | 16 GB | ✅ Comfortable |
| ROCmFPX FP6 | ~24 t/s* | ~33 t/s* | 21 GB | ⚠️ Swap at 32K |
| Stock Q6_K | ~17–23 t/s | ~65 t/s | 21 GB | ⚠️ Swap at 131K |
| Quant | HumanEval | HumanEval+ | BPW |
|---|---|---|---|
| Q6_K-MTP | 88.4% | 81.7% | 6.56 |
| Q4_K_M-MTP | 82.3% | 75.6% | 4.83 |
| ROCmFP4_FAST | 75.6% | 71.3% | 4.26 |
| ROCmFP6 | 73.8% | 67.7% | 6.50 |
Stock llama.cpp quants (Q6_K, Q4_K_M) outperform ROCmFPX quants at similar BPW — Q6_K-MTP (6.56 BPW) scores 14% higher on HumanEval+ than ROCmFP6 (6.50 BPW). The ROCmFPX quantization format trades code quality for decode speed. All quants handle math/reasoning, creative writing, and vision correctly on qualitative tests.
All quants tested on identical prompts at temperature 0.0:
Qwen3.8-27B uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers) with an integrated vision encoder. Key notes:
convert_hf_to_gguf.py → llama-quantizeconvert_hf_to_gguf.py → ROCmFPX llama-quantize (b213/b2f5829)convert_hf_to_gguf.py --mmproj --outtype bf16blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight
blk.64.nextn.attn_k.weight
blk.64.nextn.attn_k_norm.weight
blk.64.nextn.attn_norm.weight
blk.64.nextn.attn_output.weight
blk.64.nextn.attn_q.weight
blk.64.nextn.attn_q_norm.weight
blk.64.nextn.attn_v.weight
blk.64.nextn.ffn_down.weight
blk.64.nextn.ffn_gate.weight
blk.64.nextn.ffn_up.weight
blk.64.nextn.ssm_out.weight