Qwen-3.8-27B-ROCmFP4-FAST-GGUF

Qwen 3.8 27B ROCmFPX (GGUF for AMD Radeon GPUs)

This repository contains optimized ROCmFP4 and ROCmFP8 GGUF releases of Qwen 3.8 27B, custom-engineered for AMD Strix Halo (Ryzen AI Max+ 395 / Radeon 8060S) and AMD Radeon discrete GPUs.

🛠️ Unified HaloFPX Server: github.com/julianmb/halofpx (Multi-model zoo, hot-swapping, OpenAI API)
📖 Qwen 3.8 Dedicated Deep-Dive: github.com/julianmb/q38rocm (Standalone optimization report, benchmarks & tuning)

---

📦 Available GGUF Quantizations

---

🔬 Why ROCmFP4 & ROCmFP8 Improve Performance on AMD Hardware

In auto-regressive decoding, generation speed is physically bound by memory bandwidth:

• Bandwidth Math: Strix Halo's 256-bit memory controller delivers ~190–200 GB/s sustained read bandwidth. Loading an unquantized FP16 model (54.6 GB) caps generation at ~4.9 tok/s.
• 8-Bit (ROCmFP8 - 26.25 GB): Doubles memory streaming throughput to 18.96 tok/s with MTP speculation while providing lossless precision (<0.003 PPL loss vs FP16).
• 4-Bit (ROCmFP4 - 13.55 GB): Slashes memory payload by 75.2% vs FP16, raising unassisted streaming decode to 14.02 tok/s and MTP speculative throughput to 30.56 – 36.04 tok/s (with 33.80 tok/s sustained decode at K=4) with ~99% benchmark retention.

---

🧠 Optional: AMD XDNA 2 NPU Acceleration (Qwen 3.8 27B Only)

⚠️ Scope note: All NPU benchmarks below were only tested on Qwen 3.8 27B (dense, ROCmFP4_FAST).

Strix Halo features a 50 TOPS XDNA 2 NPU (/dev/accel/accel0). Extensive empirical benchmarking demonstrates:

• Sustained Decode Ceiling: 33.8 tok/s via embedded MTP (iGPU only) is the practical ceiling. The NPU does not improve sustained decode speed.
• Proven NPU Value: 1.8× faster first-token TTFT on long prompts (870 ms vs 1587 ms) via hybrid NPU-burst → iGPU handoff (scripts/run_pipeline.py).
• See docs/NPU_INTEGRATION.md for complete setup instructions.

---

📊 Performance Benchmarks (AMD Ryzen AI Max+ 395)

Measured on AMD Ryzen AI Max+ 395 (40 CU Radeon 8060S @ 2.9 GHz, 128 GB 256-bit LPDDR5X, Linux 7.0, Mesa 26.0 RADV):

---

💾 Context Scaling & Memory Footprint

Using Asymmetric TurboQuant KV cache (-ctk q8_0 -ctv turbo4):

---

📥 Quick Download

# Download ROCmFP4_FAST (13.55 GiB)
hf download julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF Qwen3.8-27B-ROCmFP4-FAST.gguf --local-dir .

# Or download ROCmFP8 (Zero-loss 8-bit precision, 26.25 GiB)
hf download julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF Qwen3.8-27B-ROCmFP8.gguf --local-dir .

---

🚀 How to Run

Option 1: Via HaloFPX (Recommended Multi-Model Server)

git clone https://github.com/julianmb/halofpx.git
cd halofpx
pip install -e .

halofpx load qwen38-27b

Option 2: Standalone via q38rocm

git clone https://github.com/julianmb/q38rocm.git
cd q38rocm

./quickstart.sh

---

🔒 Limitations & Safety

• Custom Backend: Requires the ROCmFPX toolchain (pinned build: e87d53e (213)).
• Hardware Target: Optimized for AMD Strix Halo (gfx1151) and AMD discrete Radeon GPUs.
• Base Alignment: Inherits base safety characteristics and knowledge capabilities of Qwen 3.8 27B.

---

📜 License & Attribution

• Base Model: Qwen 3.8 27B by Alibaba Cloud
• Quantization & Optimizations: Apache 2.0 License.
• Community Research: Multi-slot concurrency metrics referenced from MrWidmoreHK and kujetic.

📏 Quantization Fidelity (wikitext-2 validation)

Independent perplexity checks (llama-perplexity, ROCm0, ctx 512):

Speed is statistically identical between both builds (13.56 vs 13.64 tg128) — decode is bandwidth-bound. Requantizing any high-precision source (Q8_0/BF16, --allow-requantize) into ROCmFP4 presets is a validated recipe; avoid k-quant sources (double quantization).

🔬 Batch Scaling (why speculative decoding works here)

llama-batched-bench, tg128 per-sequence throughput, Vulkan0:

Speculative decoding converts memory-bandwidth headroom into tokens: the flat batch-cost curve is what makes MTP's ~2.5× sustained gain possible.