Qwen3.8-27B-GGUF-ROCMFPX

Qwen3.8-27B — ROCmFPX quants (Q8 full + 16 GB hybrid)

These are quantizations of unsloth/Qwen3.8-27B-GGUF (BF16) (original model: Qwen/Qwen3.8-27B, the hybrid attention/SSM qwen35 architecture with an MTP nextn head).

[!IMPORTANT] You need the ROCmFPX fork of llama.cpp (or a llama.cpp build with ROCmFPX support). These files use the experimental q8_0_rocmfpx (type 103) and/or q4_0_rocmfp4_fast (type 101) weight formats, which stock llama.cpp releases do not understand — loading them elsewhere will fail with an unknown tensor type error.

Files

Hybrid recipe (16 GB file)

Sensitive-tensor selection mirrors the tiers in Unsloth's UD-Q4_K_XL dynamic recipe (Q6_K tier = most sensitive, Q5_K = next), rebalanced onto a strict two-type q4/q8 mix to hit ~16 GB:

Everything else is q4_0_rocmfp4_fast (340 tensors, incl. ffn_gate/up/down, attn_qkv, token_embd — Unsloth keeps embeddings at Q4_K too), norms/biases stay f32 (360 tensors). The MTP head (nextn.eh_proj) is auto-protected at q8_0 by the quantizer's draft-sensitive logic.

How they were made

# pure Q8 (from the ROCmFPX fork; CPU-only build works fine for quantization)
llama-quantize Qwen3.8-27B-BF16-00001-of-00002.gguf \
  Qwen3.8-27B-Q8_0_ROCMFPX.gguf Q8_0_ROCMFPX

# 16 GB hybrid
llama-quantize \
  --tensor-type "attn_q.weight=q8_0_rocmfpx" \
  --tensor-type "attn_k.weight=q8_0_rocmfpx" \
  --tensor-type "attn_v.weight=q8_0_rocmfpx" \
  --tensor-type "attn_output.weight=q8_0_rocmfpx" \
  --tensor-type "attn_gate.weight=q8_0_rocmfpx" \
  --tensor-type "ssm_out.weight=q8_0_rocmfpx" \
  --tensor-type "^output.weight=q8_0_rocmfpx" \
  Qwen3.8-27B-BF16-00001-of-00002.gguf \
  Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf Q4_0_ROCMFP4_FAST

Usage

This quant is text weights only. Qwen3.8 is multimodal — for vision support, pair it with the mmproj file: this repo includes mmproj-F16.gguf (mirrored from unsloth/Qwen3.8-27B-GGUF, which also offers mmproj-BF16.gguf).

# build ROCmFPX for your GPU (see the repo README; e.g. Strix Halo):
env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh

./build-strix-rocmfp4/bin/llama-cli \
  -m Qwen3.8-27B-Q8_0_ROCMFPX.gguf --mmproj mmproj-F16.gguf \
  -dev Vulkan0 -ngl 999 -fa on --jinja

Benchmarks

[!NOTE] All Benchmarks were run on an AI max 395+ with 128GB RAM

Quality comparison vs BF16 source (perplexity / KLD): TBD.

Attribution & license

• Quantized from: unsloth/Qwen3.8-27B-GGUF (BF16 shards); sensitivity tiers referenced from their UD-Q4_K_XL
• Original model: Qwen/Qwen3.8-27B
• License: Apache-2.0 (inherited)
• Quant formats by the ROCmFPX project