Qwen3.8-27B-exl3-8.00bpw

Qwen3.8-27B EXL3 8.0bpw

What this is

This is an 8.0bpw EXL3 quantized derivative of Qwen/Qwen3.8-27B. It exists because, at the time of packaging, no public 8bpw EXL3 release was available: the reference Qwen3.8 EXL3 repository topped out at 6.00bpw. That is the main reason to choose this release over the existing public branches.

The language-model weights were transformed from the upstream BF16 checkpoint into EXL3. The model's MTP component and BF16 vision tower were retained. This card replaces the upstream card so that the modification, serving constraints, and local measurements are explicit.

Quantization details

These values come from the release's own quantization_config.json, not from a target recipe or filename:

The exact exllamav3 build above loaded the artifact successfully. Compatibility with older exllamav3 releases is not claimed. Calibration used the converter's packaged stock corpus with no custom calibration source.

How to load

For TabbyAPI, place the downloaded branch below your model directory and use the following per-model settings. The model path is deliberately a placeholder; the remaining values come from the configuration used for the measurements in this card.

model:
  model_dir: /path/to/models
  model_name: Qwen3.8-27B-exl3-8.0bpw
  backend: exllamav3
  tool_format: qwen3_coder
  reasoning: true
  reasoning_start_token: "<think>"
  reasoning_end_token: "</think>"
  max_seq_len: 212992
  cache_mode: "4,4"
  gpu_split_auto: true
  autosplit_reserve: [500, 96]
  chunk_size: 2048
  max_batch_size: 1

draft_model:
  draft_mode: mtp

draft_model is a top-level TabbyAPI block, not a child of model. Sampling is supplied by the request or caller: retain temperature: 1.0, top_p: 0.95, and top_k: 20. Image input also requires TabbyAPI's vision option; leaving it off keeps the retained tower out of VRAM.

For plain exllamav3, the included argument-driven generator is both a load check and a reproducible sample harness:

python scripts/generate_exl3_release_samples.py \
  --model_dir /path/to/Qwen3.8-27B-exl3-8.0bpw \
  --output-dir ./catbench \
  --model-label Qwen3.8-27B-exl3-8.0bpw \
  --bitrate-label 8.00bpw \
  --cache_size 16384 --cache_quant 4,4 --mtp \
  --require-reasoning-split

The script applies the Hugging Face chat template with thinking enabled, uses the recommended sampling triple, enables model-owned MTP, and rejects truncated or unclosed reasoning samples. Its model and output paths are arguments, so it can be reused unchanged for another EXL3 quant.

MTP is preserved

The model-owned Multi-Token Prediction component is present: the artifact index contains 39 mtp.* tensors, including eight EXL3 linears stored at 4bpw. TabbyAPI can use it for self-speculative decoding without a separate draft model:

draft_model:
  draft_mode: mtp

At 196,608-token configured context, using temperature: 1.0, top_p: 0.95, and top_k: 20, an interleaved three-run A/B on the same multi-step agentic task measured:

The MTP-on median was 1.87× faster, all six tasks passed, and 2,576 of 3,997 drafted tokens were accepted (64.45%). MTP added 1,836 MiB at the measured peak, so its speedup is not free.

Vision tower

The complete vision tower is retained as 333 BF16, unquantized tensors totaling about 858 MiB. It supports the base model's image/video architecture and accounts for disk space even in a text-only deployment.

TabbyAPI does not load the tower unless vision is enabled. Text-only users therefore pay the disk cost, not the vision tower's VRAM cost.

Serving guidance — do not use greedy decoding

[!WARNING] Do not serve this model greedy. Use temperature: 1.0, top_p: 0.95, and top_k: 20 for reasoning and agentic traffic.

The warning is based on an identical long-reasoning prompt across the available backends:

The EXL3 build did not reproduce the GGUF repetition collapse: its 0.9204 ratio and 16× worst repeat stayed outside the predeclared collapse threshold. Greedy still consumed the full 8,192-token generation budget without closing the reasoning trace or producing a visible answer, so it remains an unusable serving regime for long reasoning.

Short tool-call probes passed under both greedy and vendor sampling. A green short tool sweep therefore does not clear greedy serving; it does not exercise the long-turn failure.

Measured performance and VRAM

All GPU measurements below used an NVIDIA RTX 4070 12 GB plus RTX 3090 24 GB, a nominal 36,864 MiB combined. Rates are specific to the named task, prompt depth, sampling regime, MTP setting, and software build; they are not generic maximum-throughput claims.

The final serving ceiling was separately validated at 212,992 tokens with MTP enabled. A 15-cell retrieval grid was exact through 206,862 raw prompt tokens. Although a 229,376-token cache could start, real prefills failed above 212,992 on this hardware; the base model's 262,144-token native declaration is not a claim that this quant can serve that window within the measured VRAM budget.

The only controlled cold figure is the 30.83-second, 196,608-context, MTP-off measurement. Loads of the final 212,992-context/MTP-on configuration were observed with a warm page cache, so they are deliberately not reported as cold-start results.

Verification collateral

The BF16 checkpoint remained available, so quantization fidelity was measured directly with the pinned exllamav3 1.4.2 qbench rather than estimated from bitrate:

PPL r-100 uses 100 non-overlapping WikiText2 test rows of 2,048 tokens with stride 2,048 and no chat wrapper: 204,700 next-token targets. KL compares 204,800 quantized logit positions directly with the cached BF16 distributions. The optional BF16 self-noise-floor pass was not run, so mean KL is unadjusted. Its median was 0.00036012 and p90 was 0.00115159. The GiB column follows the reference CSV convention and reports safetensors shard bytes, not runtime VRAM. The publishable source table is Qwen_Qwen3.8-27BMeasured.csv.

Three fixed-prompt samples are under catbench/. They use the vendor sampling triple and model-owned MTP, close their reasoning traces, produce visible answers, and stop naturally. The short explanation is 416 tokens (distinct-12-gram 0.992593), the concurrency review is 2,434 tokens (0.924887), and the long constructive proof is 3,319 tokens (0.983676, worst repeated 12-gram 4×). The argument-driven generator and reusable qbench helpers are included under scripts/.

Limitations

• Correctness was measured with a small agentic suite that both this model and the comparison model saturated: 9/9 versus 9/9 on the three-task mini-benchmark and 3/3 versus 3/3 on the approximately 85k-token deep probe. The instrument was too easy to resolve fine capability differences; the tie does not establish equal capability.
• Under the established correctness-first decision rule, this model's head-to-head outcome against qwen3.6-27b-8bpw-exl was lose: the py-hello latency ranges were disjoint in the comparison model's favor, while the other two mini-task ranges overlapped, and this model took 1.353–1.470× as long on the three deep runs. It did, however, displace the separately measured thinking alias, with 9/9 correctness on both sides and non-overlapping speed wins on all three mini tasks.
• The KL/PPL result is a single WikiText2 distribution-fidelity measurement, not a reasoning, tool-use, vision, or instruction-following benchmark. qbench's optional BF16 self-noise-floor pass was not run; the reported mean KL is direct and unadjusted.
• The retained vision tensors were inventoried, but the performance and correctness results above are text/agentic measurements, not a visual-quality evaluation.
• The containment check observed one successful stochastic agentic run with no out-of-sandbox file changes. That pass cannot prove that a rare future failure is impossible.
• The exact exllamav3 build named above was tested. Other exllamav3, TabbyAPI, driver, GPU, context, cache, batching, or sampling combinations may perform differently.

License

This quantized derivative is distributed under Apache License 2.0, matching the base model. The included LICENSE file carries the applicable terms and notices. Qwen names and marks remain the property of their respective owners; this release does not imply upstream endorsement.