Qwen3.8-27B-UD-Q4_K_XL-AWQ-MTP-mlx

Qwen3.8-27B-UD-Q4_K_XL-AWQ-MTP-mlx

Q4_K_XL quantization of Qwen/Qwen3.8-27B for MLX on Apple silicon, with AWQ imatrix pre-scaling and the MTP head kept inline.

AWQ here means activation-aware pre-scaling applied before MLX affine quantization — the same sense as mlx-community/*-4bit-AWQ. The stored format is standard MLX affine, not AutoAWQ/GPTQ packing, so this will not load in vLLM or autoawq. MTP means the multi-token-prediction head ships inside the main shards rather than as a separate drafter.

Read the BPW, not the name

This model is 6.02 bpw, not 4 bpw. The Q4_K_XL label follows the Unsloth naming convention, where --q-bits sets the base class of a mixed-precision recipe rather than the average. Higher-precision classes then sit above that base and dominate the file. The real distribution:

Sidecars are the per-group scales and biases required by affine quantization at group size 64; they carry no parameters but do occupy space.

Which weight gets which width

Every quantized tensor, by module and resolved bit width (370 explicit per-layer overrides; anything not overridden inherits the 4-bit top-level default):

Attention o_proj and GDN out_proj are pinned high because AWQ cannot correct them (no preceding norm to fold into), so they carry uncompensated quantization error and are given extra width instead.

Size it against your RAM using the 20.89 GB figure plus KV-cache headroom, not against what Q4_K_XL implies elsewhere.

Sources

The unsloth recipe is mlx-node's reimplementation of Unsloth's per-layer class map. It is not an Unsloth-produced artifact and carries none of Unsloth's calibrated NVFP4 global scales, W4A4/W8A8 activation execution, or calibrated FP8 KV-cache scales. No numerical or performance parity with upstream Unsloth is claimed.

The imatrix was produced by AtomicChat and published alongside their Qwen3.8-27B quantization comparison; its author describes it as calibrated on their public corpora. Inside the GGUF the corpus is recorded only as the producer's local path (/eval/calib_train.txt), so the exact composition is not asserted here — see the linked discussion and collection for their methodology.

How it was built

mlx convert \
  --input  <Qwen/Qwen3.8-27B snapshot> \
  --output Qwen3.8-27B-UD-Q4_K_XL-AWQ-MTP-mlx \
  --model-type qwen3_5 \
  --quantize \
  --q-bits 4 \
  --q-group-size 64 \
  --q-recipe unsloth \
  --imatrix-path qwen3.8-27b-atomicchat-imatrix.gguf \
  --q-mtp off

--q-mtp off retains the MTP tensors inline in the main shards, unquantized. There is no mtp-drafter/ directory and no mtp.safetensors sidecar; loaders that expect a split drafter will not find one here.

AWQ pre-scaling amplifies important weight columns using the imatrix and folds the inverse into the preceding norm, so the reparametrization is output-preserving. It runs on MLP gate/up/down, attention q/k/v, and GatedDeltaNet in_proj_*. Attention o_proj and GDN out_proj are deliberately excluded — their inputs come from attention output, not from a norm, so there is no norm to fold into.

Architecture

Usage

Requires an MLX runtime with qwen3_5 support (hybrid GatedDeltaNet + full attention). Stock mlx-lm does not implement this architecture.

mlx chat --model Qwen3.8-27B-UD-Q4_K_XL-AWQ-MTP-mlx

Performance

Measured on Apple M3 Max (30-core GPU) under oMLX, tokens/sec:

All three variants, same machine:

Prompt processing is compute-bound and sits at 154–160 tok/s regardless of quant width. Token generation is memory-bandwidth-bound and orders by file size at 1k. At 4k the Q4 and Q5 figures cross over; these are single runs taken about a minute apart on one machine, so differences of ~1 tok/s are within run-to-run and thermal noise and should not be read as a real ordering.

Verification status

Structure, quantization metadata, and BPW in this card were measured directly from the safetensors headers and config.json of these files. The model loads and generates — the throughput above is from a real run. Output quality has not been benchmarked: no perplexity, and no downstream eval against the BF16 base or other quantizations.