Qwen3.8-27B-Uncensored-int4-AutoRound

Qwen3.8-27B-Uncensored INT4 AutoRound (MTP-preserved)

An INT4 (W4A16) quantization of an uncensored/abliterated 27B dense Qwen3.8 variant, built for fast local serving on Intel Arc Pro B70 GPUs with vLLM XPU.

Two deliberate choices distinguish this checkpoint from a stock INT4 quant:

1. The MTP (multi-token prediction) head is preserved and quantized, not stripped. Speculative decoding works out of the box with vLLM's qwen3_5_mtp method — no separate draft model or weights surgery needed.
2. The linear-attention in_proj_a / in_proj_b projections are kept in FP16. Quantizing those two projection families to INT4 g128 measurably degrades exact-answer behavior (e.g. deterministic code-evaluation prompts return wrong values). Keeping them FP16 costs a few percent of decode bandwidth and preserves target-model quality. Verify any other INT4 build of this model against a deterministic canary before trusting it.

Quantization

The checkpoint is directly servable as GPTQ INT4 (Marlin / XPUwNa16LinearKernel path in vLLM).

Measured performance (Intel Arc Pro B70)

Cold-response measurement discipline: fixed realistic prompt suite, each prompt run once, prefix/prompt caching disabled, cached_tokens=0 on every request; primary metric is the median inter-token rate over tokens 1-100 after TTFT.

Quality gates on this exact checkpoint: arithmetic, deterministic code evaluation, copy, factual, JSON-schema, long-context needle, and repeat-stability checks all pass; all exact canaries are identical to the FP16-head reference.

Notes on interpretation: warmed, repeated-prompt, same-shape benchmarks of this stack read much higher (80+ on a single card); those numbers describe repeat traffic, not fresh requests.

Serving (vLLM XPU)

vllm serve /path/to/this/model \
  --quantization gptq \
  --dtype float16 \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prompt-tokens-details \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

• Single-GPU serving needs no tensor parallelism flags; the model fits in one 32 GB B70 with FP16 KV at moderate context.
• FP8 KV cache (--kv-cache-dtype fp8) works and is the capacity option for 128K context on one card; FP16 KV decodes slightly faster.
• The MTP head accepts num_speculative_tokens 1-4. Depth 3 was the best measured trade-off on fresh-response workloads; depth 4 can win on highly predictable content (acceptance is workload-dependent — check vllm:spec_decode_num_accepted_tokens_total against spec_decode_num_draft_tokens_total on your own traffic).
• Optional experimental lever: an INT8-quantized LM head (W8A8) served via a patched vllm-xpu-kernels build adds ~15% decode on this model class; the stock FP16 head path is what the 81.8/57.5 rows above use.

Chat template

Ships the stock Qwen3.8 chat template (chat_template.jinja; reasoning and tool-call blocks follow the base model's conventions). Note for serving: vLLM's thinking_token_budget (or an equivalent hard cap) is the reliable way to bound reasoning length; prompt-level effort instructions are a hint the model can ignore on hard prompts.

Limitations

• This is an uncensored model: it will answer prompts that safety-tuned models refuse. You are responsible for how you use it; review generated content before relying on it.
• INT4 quantization trades some quality for speed and size. The preserved FP16 projection families mitigate the worst regression we measured, but outputs are not bit-identical to the BF16 original, and stylistic casing differences on short answers have been observed.
• Benchmarks above are single-stream; concurrent serving aggregates higher.