Qwen3.8-27b-5.22bpw-exl3

Qwen3.8-27B — EXL3 4.5bpw (HQ)

EXL3 quantization of Qwen3.8-27B (Qwen3.5-architecture, 27B total / 64 layers, 262144 ctx, native vision) at an overall 4.5 bits per weight, high-quality mode, for use with ExLlamaV3 / TabbyAPI.

Two variants are provided — same language model, different vision tower precision:

The q6-vision variant saves an additional ~316 MiB (331,809,546 bytes) over the bf16-vision variant, on top of the ~69 GiB already saved by quantizing the language model to 4.5bpw from the 51.8 GiB BF16 source.

Quantization recipe

-b 4.5 -vb 6 -hb 8 -mb 6 -hq -cb mul1

• -b 4.5: target average 4.5 bpw for the budgeted language-model weights (attention/MLP projections inside the decoder layers), allocated per-layer by EXL3's Hessian-aware bit assignment
• -vb 6: vision tower at 6 bpw (this variant) — see vision-bf16 for the 16bpw/unquantized alternative
• -hb 8: LM head at 8 bpw
• -mb 6: misc/auxiliary modules (MTP draft head) at 6 bpw
• -hq: high-quality mode (clamps every module to at least its calibrated minimum bitrate instead of letting the bit-budget allocator trade quality away for a lower average)
• -cb mul1: calibration batching mode

Total effective bits-per-weight

-b 4.5 is the target for the budgeted decoder weights only — it excludes embed_tokens (always kept BF16) and lm_head (fixed at 8 bpw), both of which are large enough to pull the whole-file average up. Computed directly from quantization_config.json's per-tensor storage manifest (language model + head + embeddings, 26.89B params, 17.56 GB stored):

So while the LM's core decoder weights land almost exactly on the 4.5 target, the whole-file effective average is ~5.22 bpw once embed_tokens and lm_head are folded in — this is the number to quote for an honest overall compression ratio. (Vision tower + MTP draft head are quantized separately at 6 bpw each and aren't itemized in this same manifest; they're a small fraction of total params and don't move the headline number much.)

⚠️ Required patch to load this model (q6-vision variant only)

The vision tower's MLP layers (intermediate_size=4304, not a multiple of 128) are quantized using EXL3's blockwise Hadamard transform, which requires both weight-matrix dimensions to be multiples of 128. To make that possible, the MLP is zero-padded to 128 before quantization (mathematically inert through GELU, since GELU(0)=0).

As of exllamav3 v1.4.2, the stock architecture definition does not know about this padding — it still declares the vision MLP with pad_to=1, so a vanilla install will fail to load this variant with:

RuntimeError: unpacked and packed have incompatible shapes

Until this is upstreamed, apply this one-line patch to your exllamav3 install before loading:

exllamav3/architecture/qwen3_vl.py — in the vision block's MLP(...) constructor, remove the line:

pad_to = 1,

That's the only change needed; the default pad_to=128 then matches how these weights were actually quantized. This patch is not required for the vision-bf16 variant, which uses the stock unpadded path.

Usage

Works with TabbyAPI (exllamav3 backend) like any other EXL3 model — set vision: true in config.yml and point model_dir/model_name at this repo. Verified working: loads cleanly and correctly interprets image content through the chat completions API (tested against a synthetic image with distinct shapes/colors/text, description matched exactly).

Tested performance (single RTX 3090, 24GB)

150,000-token context, Q8 KV cache, native MTP self-speculative decoding (this model ships an MTP draft head, mtp_bits: 6) at draft depth 2 with its own Q8 cache, max_batch_size: 1, fully GPU-resident (no system-RAM spillover) — ~22.5 GB VRAM used. Benchmarked (3 runs, 400 tokens each, greedy decoding):

Note: this model's hybrid linear/full-attention architecture defaults to max_batch_size: 4 in TabbyAPI (vs. 128 for plain transformers), since recurrent-layer state costs more per concurrent sequence. For single-stream use, explicitly setting max_batch_size: 1 avoids reserving CUDA-graph buffers for unused batch shapes — this alone freed enough VRAM to fit MTP depth-2 drafting at the full 150k context on a 24GB card.

Files

Standard EXL3 layout: config.json, quantization_config.json, model.safetensors.index.json, safetensors shards, tokenizer/chat-template files.