Qwen3.8-27B-abliterated-GPTQ-Pro

Qwen3.8-27B-abliterated-GPTQ-Pro

MTP performance

Measured with vLLM on 2× RTX 3090 GPUs using tensor parallelism, a 4K context, greedy decoding, and 256 generated tokens.

<!-- polished-overview:start -->

Overview

Qwen3.8-27B-abliterated-GPTQ-Pro is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

The repository name identifies a behavior-modified or reduced-filtering lineage. That label describes the source or conversion history; it is not a guarantee of unrestricted behavior in every prompt or runtime. Test outputs carefully before sharing or deploying them.

At a glance

What is included

• *.safetensors (5 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• processor_config.json
• chat_template.jinja
• quantize_config.json
• Additional configuration, tokenizer, processor, or shard files (18 visible artifacts total)

Quick start

vLLM (documented configuration)

vllm serve groxaxo/Qwen3.8-27B-abliterated-GPTQ-Pro \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --dtype bfloat16 --language-model-only \
  --max-model-len 260000 --gpu-memory-utilization 0.92 --max-num-seqs 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

INT4 GPTQ-Pro quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, with the MTP (multi-token prediction) draft head preserved and served natively by vLLM's speculative decoding.

This is the calib-128 build, using 128 calibration samples.

Quantization details

Per-module quantization loss (quant_log.csv, included): mean 8.2e-5, max 4.9e-4. Damping 1.49–8.94 (mean 3.33).

vLLM MTP speed test (measured, 2× RTX 3090, TP=2)

vLLM 0.24.0, max_model_len=4096, greedy, 256 generated tokens, prompt about MTP itself. Greedy decoding → measured draft acceptance at position 0: 100%.

Single-GPU note: on one RTX 3090 (23.56 GiB) the MTP drafter does not fit alongside this checkpoint — target ~21.5 GiB + drafter ~2.4 GiB exceeds the card. Single-GPU no-MTP baseline for reference: 42.9 tok/s single-stream, ~273 tok/s at 8-way concurrency.

Production serving (2× RTX 3090, TP=2, vLLM 0.24.0)

Deployed behind a router serving real chat traffic (temp 0.7):

Measured single-stream speeds (temp 0.7 unless noted):

Findings:

• Use draft-3, not draft-1. Draft-1 wins only under greedy decoding (100% position-0 acceptance); at real sampling temps it is slower (~77% acceptance → ~1.77 tok/step) than draft-3 (~2.76 tok/step).
• Speculative decoding degrades under batching: 2 concurrent streams yield 56.9 tok/s combined vs 84 sequential. max_num_seqs=1 maximizes single-stream speed; 2 trades aggregate throughput for simultaneous admission.
• bf16 KV halves KV pool vs FP8 (331,750 tokens) but no measurable quality/speed penalty in chat use.
• min_p and logit_bias do not work with speculative decoding (vLLM limitation).

⚠ Critical flag for multi-GPU RTX 3090 (PCIe, no NVLink): pass --disable-custom-all-reduce. vLLM's custom allreduce kernel silently corrupts tensor-parallel output on this hardware class — endless !!!!... with zero errors. Verified by A/B: identical checkpoint and prompts, broken without the flag, byte-identical correct output with it. NCCL_P2P_DISABLE=1 does not fix this. Affects any TP-served model on such boxes, not just this quant.

Usage

vllm serve groxaxo/Qwen3.8-27B-abliterated-GPTQ-Pro \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --dtype bfloat16 --language-model-only \
  --max-model-len 260000 --gpu-memory-utilization 0.92 --max-num-seqs 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Without MTP, serve like any GPTQ checkpoint (single 24 GB GPU works with --max-model-len 4096 --gpu-memory-utilization 0.92 --max-num-seqs 32).

Note: min_p and logit_bias do not work with speculative decoding (vLLM warning).

Acknowledgements

• huihui-ai for the abliterated source model
• Qwen — Qwen3.8-27B
• GPTQModel / ModelCloud for GPTQ-Pro