qwen3.8-27b

Qwen3.8-27B MQ V2 ladder for hipfire

Text-only Qwen3.8-27B artifacts for hipfire, the Rust-native LLM inference engine for AMD GPUs.

This repository contains the admitted MQ3V2 through MQ6V2 product ladder, the MQ4L (MQ4V2-Lloyd) tiers, and matching DFlash drafters. MQ4V2 supersedes MQ4V1 as hipfire's canonical mq4 product. Artifacts remain self-describing by HFQM quant-type ID; old MQ4V1 files are still readable but are not the current product.

Product filenames

Each bit-width has exactly three tiers:

• mqN-xt: embeddings protected; lm_head and recurrent outputs remain at the base codec.
• mqN: embeddings and lm_head protected.
• mqN-pro: recurrent/SSM outputs also lifted. MQ3 Pro uses ssm_out:mq6v2 to stay inside the 3.x model-bpw class.

mq4-xt was re-issued on 2026-09-15: the original upload was missing its AWQ activation-scale sidecars (WT2 KLD 0.057449); the current file carries them (0.048190). mq4 and mq4-pro are unchanged (byte-identical to their AWQ'd originals).

MQ4L: MQ4V2-Lloyd tiers (qt=52)

mq4l-xt, mq4l and mq4l-pro are the MQ4V2 tiers re-quantized with a per-tensor 16-level Lloyd-Max codebook in place of the uniform 4-bit grid. The container is byte-compatible with MQ4V2 (same 136 B groups, same AWQ sidecars, same size); each 4-bit tensor additionally carries a 64-byte <tensor>.lloyd_levels sidecar. Teacher KLD drops by ~15% at identical bits per weight (xt 0.048 -> 0.041, base 0.039 -> 0.035, pro 0.032 -> 0.028).

Requirements: hipfire with MQ4L support (quant type 52; hipfire --version >= the release that ships it). Runtime coverage today: RDNA4 (gfx1201, e.g. Radeon AI PRO R9700) prefill + decode. RDNA3 (gfx1100 / Strix Halo gfx1151) decode is supported; batched prefill is not yet (the loader refuses rather than falling back). Speed on gfx1201 vs the uniform tier: decode -1%, prefill -1%. Tags: qwen3.8:27b-mq4l-xt, qwen3.8:27b-mq4l, qwen3.8:27b-mq4l-pro. The MQ4V2 DFlash drafter pairs with the MQ4L targets unchanged.

MQ2V2 was fully measured but is deliberately not published as a product: all three tiers produced catastrophic WT2 KLD around 12–13 and degraded/repetitive text.

DFlash drafters

Sizes are decimal GB, matching Hugging Face.

The MQ4V2 control drafter won the matched decode screen for every admitted MQ3/MQ5/MQ6 target tier and is therefore the recommended default controller. The same-bit drafts remain available for explicit size/codec experiments.

DFlash is opt-in:

hipfire pull qwen3.8:27b
hipfire pull qwen3.8:27b-draft-mq4
hipfire run --spec dflash \
  --model-draft ~/.hipfire/models/qwen38-27b-dflash-mq4.hfq \
  qwen3.8:27b "Design a streaming Rust file deduplicator."

Usage

# Canonical MQ4V2 base tier
hipfire pull qwen3.8:27b
hipfire run qwen3.8:27b "Design a streaming Rust file deduplicator."

# Explicit ladder cells
hipfire pull qwen3.8:27b-mq3-xt
hipfire pull qwen3.8:27b-mq4-pro
hipfire pull qwen3.8:27b-mq6

Runtime defaults

Every non-draft Qwen3.8 ladder tag receives the same registry tag policy:

Quality and performance evidence

Teacher KLD

The WT2 KLD values in the product table use 24 scored chunks against the pinned WT2 teacher with Q8 KV and prefill scoring. They measure quantization quality rather than runtime speed. A second 24-chunk validation-selection fixture (v6sel) independently preserved the same tier ordering. MQ2V2 was rejected before publication because its WT2 KLD was 12.44–13.24.

HumanEval pass@1

All 36 product × decode-mode arms ran the full 164-task HumanEval suite on gfx1201: one stateless native-serve request per task, greedy sampling, thinking off, Q8/VMM KV, and an 81,920-token output ceiling. DFlash rows use the recommended MQ4V2 controller; MTP rows use the matching product sidecar. HE/2 + HE/3 is the explicit smoke pair used throughout ladder bring-up.

DFlash controller screen

This fresh-process gfx1201 screen compares each target with its same-bit draft and with the MQ4V2 control draft. Decode is tok/s; $\tau$ is emitted tokens per verification cycle. MQ4V2 won or tied every admitted target, so it is the registry default while all four drafters remain downloadable.

Native throughput

The full promoted product matrix on Radeon RX 7900 XTX (gfx1100) uses fixed-shape hipfire bench --matrix with Q8/VMM KV, --spec off, --backend noslots, --workload stateless, 5 measured runs, and 3 warmups. TG128@128 generates 128 tokens from an initial context of 128 tokens. The MQ4 rows are the 2026-08-24 post-multi-wave screen; MQ5 and MQ6 were refreshed on 2026-08-26 at runtime commit 363e93d77.

These are fixture-bound medians, not performance floors. The complete five-sample arrays, artifact hashes, and binary hashes are preserved in the dated historical checkpoint.

An earlier base-tier screen at runtime commit 9b6a83ae8 used 3 measured runs, 5 warmups, and a different PP2048/single-token decode fixture:

The gfx1100 MQ3V2 batched WMMA route remains deliberately unpromoted after a device-loss incident in DFlash verification; AR uses the proven row-wise path. On gfx1201, the full HumanEval AR runs above measured median prefill/decode throughput of 514.35/37.3 tok/s (mq3), 623.5/33.1 (mq4), 470.2/28.1 (mq5), and 582.0/25.1 (mq6). HumanEval prompt lengths vary, so those gfx1201 medians are not fixed-shape PP512 measurements and are reported separately.

Wire format

MQ V2 uses an 8-byte dual-FP16 affine header per 256-weight group: one scale/zero pair for weights 0–127 and one for 128–255. Wire IDs are MQ4V2 qt44, MQ6V2 qt47, MQ5V2 qt48, and MQ3V2 qt49.

License and attribution

The upstream Qwen3.8 weights are Apache-2.0 licensed. These quantized artifacts were produced for hipfire using its MagnumQuant FWHT-rotated quantization family. See the upstream model card and hipfire provenance.