Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings

Qwen3.8-27B W4A16 AutoRound with INT8 Head and Embeddings

This checkpoint starts from dbirks/Qwen3.8-27B-W4A16-AutoRound and applies the serving recipe published by syv-ai/qwen38-27b-rtx3090.

The language-model linear layers remain the original AutoRound W4A16 compressed-tensors quantization. The two large untied vocabulary matrices are additionally quantized to symmetric INT8 with group size 128:

• lm_head: INT8 group-128, relative round-trip error 0.0064
• model.language_model.embed_tokens: INT8 group-128, relative round-trip error 0.0056

This saves approximately 2.6 GB of weight memory compared with the source W4A16 checkpoint, leaving more VRAM for KV cache and recurrent state pages on 24 GB GPUs.

Provenance

• Base model: Qwen/Qwen3.8-27B
• Base model revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
• Source quant: dbirks/Qwen3.8-27B-W4A16-AutoRound
• Source checkpoint revision: 1f05c441c4e64ae0549de44fa9ea5a6d43610314
• Transformation recipe: syv-ai/qwen38-27b-rtx3090
• Recipe revision: e681a8f6b190cca1f86d58b8b333b64d014994d7
• Output format: compressed-tensors pack-quantized
• Body: symmetric INT4 W4A16, group size 128
• Vocabulary matrices: symmetric INT8, group size 128

The exact transformation scripts and vLLM patch used are included in this repository.

Important vLLM requirement

The INT8 token embedding table requires a small vLLM patch. Stock vLLM 0.27.1 contains the dequantization kernel but does not pass the quantization config to Qwen3.5/Qwen3.8's VocabParallelEmbedding constructors.

Apply the included patch to the installed vLLM package:

patch -p1 -d /path/to/site-packages/vllm \
  < vllm-qwen3_5-embed-quant.patch

The patch was written and verified against vLLM 0.27.1. Revalidate it after upgrading vLLM. Without this patch, the checkpoint is not expected to load correctly.

Text-only serving on a 24 GB GPU

The following profile mirrors the batch-oriented syv-ai recipe:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve XReyRobert/Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings \
  --served-model-name qwen3.8-27b-autoround-int8-head-embeddings \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --max-model-len 150000 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.972 \
  --async-scheduling \
  --compilation-config '{"max_cudagraph_capture_size":64,"custom_ops":["+rms_norm","+silu_and_mul"]}' \
  --trust-remote-code

--language-model-only avoids loading the vision tower. The published syv-ai measurements use an RTX 3090 and describe 150k as the conservative deployment context; the reported single-request ceiling after requantization is around 195k. Actual capacity depends on the runtime version, CUDA allocator state, concurrency and prompt shape.

For single-user speculative decoding, use the MTP profile documented in the source recipe and ensure the second hunk of the included patch is applied.

Sampling and thinking control

Use the source model's recommended sampling parameters:

Qwen3.8 supports reasoning_effort=low, medium, or xhigh; xhigh is the source-model default. For agentic sessions, enable thinking and preserve prior reasoning through the chat template:

{"enable_thinking": true, "preserve_thinking": true}

Keep the combined input, retained reasoning, and generated output inside the served context limit. The 150K profile above is smaller than the model's native 262,144-token position limit because it is sized for one 24 GB GPU.

Reproduction

Starting from the pinned source revision:

python quant_lm_head.py /path/to/checkpoint
python quant_embed.py /path/to/checkpoint

The scripts rewrite the relevant safetensors shards, update model.safetensors.index.json, and add dedicated compressed-tensors config groups for lm_head and embed_tokens.

Validation performed

• Safetensors/index consistency: 2,003 indexed tensors, 2,003 present
• Missing indexed tensors: 0
• Extra tensors in indexed shards: 0
• lm_head relative round-trip error: 0.0064
• embed_tokens relative round-trip error: 0.0056

Terminal-Bench 2.0 sanity check

The published artifact was exercised with Terminus 2 on three tasks (openssl-selfsigned-cert, headless-terminal, and count-dataset-tokens) using one RTX 3090, one concurrent trial, 32 CPU cores, 48 GiB RAM, and a 30-minute task timeout.

The xhigh timeout occurred after active generation and is retained as an observed benchmark outcome even though the verifier reward was already 1 for that task. Parser warnings indicate extra or discarded model text around tool actions; they do not automatically imply an incorrect tool call.

This is a serving and agent-path smoke check, not a broad quality ranking. Three tasks are too few for comparative accuracy claims. Refer to the source AutoRound model card for body-quantization evaluations and to the syv-ai repository for the original INT8 head/embedding serving benchmarks.

Credits

• Qwen team for Qwen3.8-27B
• dbirks for the W4A16 AutoRound checkpoint
• syv-ai for the INT8 head/embedding transformation and vLLM serving recipe