Qwen3.8-27B-AutoRound-W4A16-G128

Qwen3.8-27B-AutoRound-W4A16-G128

27B model. INT4 weights. One 32 GB B65.

An Intel Arc Pro B65/B70-targeted AutoRound quantization for vLLM XPU serving, distributed in safetensors with GPTQ-compatible packing. Quantized directly from the original Qwen3.8-27B BF16 checkpoint, with pinned source and creation provenance below.

Tested on B65 using patched vLLM XPU. B70 is a deployment target, not a tested result.

At a glance: one B65

All headline rates are for one 32 GB Intel Arc Pro B65. Single-request decode rates exclude prefill; concurrent throughput is the aggregate over the streams' common decode interval. The full-context workload took 13.3–13.6 minutes to prefill without cache hits. Full measurements and timing definitions appear below.

Independent quantization and qualification by Ryan Weddle. Original model by Qwen; tooling by Intel AutoRound. Not an official Qwen or Intel release.

Artifact

HF sidebar: the “6B” count reflects packed storage, not the model's logical 27B weights. INT4 weights are packed into I32 tensors; BF16 holds preserved weights and F16 holds quantization scales.

Embeddings, the output head, vision, MTP, normalization, and selected recurrent-attention projections remain unquantized. AutoRound identifies the quantization method; GPTQ-compatible packing identifies the export format.

MTP is stored in BF16. Preserve model_extra_tensors.safetensors and its associated configuration. The INT4 MTP/draft-head treatment tested during inference is a separate runtime operation.

Creation provenance

Quantized directly from the pinned BF16 source.

• Method: AutoRound 0.16.0.dev136+gce90add6, BF16 calibration activations.
• Calibration: NeelNanda/pile-10k, 128 samples × 2,048 tokens.
• Tuning: 200 iterations, seed 42, batch size 8, gradient accumulation 1.
• Execution: Torch compilation and low-memory modes enabled; NVIDIA A100 80GB.
• Software: PyTorch 2.6.0+cu124, Transformers 5.8.0, Triton 3.2.0.

<details> <summary>Exact revisions and calibration checksum</summary>

• Model: 1d4bf0f2ff60
• AutoRound: ce90add61e93
• Dataset: 127bfedcd504

Full revisions are encoded in the links. Recorded calibration-token file SHA-256:

338b5b962c754e137501d39859632e7e3175ae1f0c35ed57c9e2c4d46080e86d

</details>

The export metadata abbreviates AutoRound's version to 0.16.0; the development version and commit above identify the creation software. The preserved transfer receipt records successful SHA-256 verification of all 76 export files.

B65 serving qualification

Hardware and runtime

All throughput rates below are for one Intel Arc Pro B65 with 32 GB GDDR6, not a combined multi-GPU rate. Intel's B65 datasheet specifies 608 GB/s memory bandwidth and 197 peak dense INT8 TOPS. TOPS is a vendor XMX compute rating, not measured W4A16 throughput.

Tested with patched vLLM XPU and LMCache, FP16 target compute, native W4A16 kernels, FP8 KV, and PIECEWISE compilation. MTP-enabled runs used separate runtime INT4 MTP/draft-head handling.

Decode throughput

Rates are tokens/second. Single-request rates measure streamed decode, excluding prefill; the four-request row is per-GPU aggregate common-overlap decode, not per-request throughput.

Short-context results used 2,048 generated tokens in the September 8, 2026 run. Long-context results used 512 generated tokens and geometric means over two counterbalanced September 9 rounds. Different workloads and runs make this a serving summary, not a controlled context-scaling experiment.

Recommended for these tested workloads: MTP4 at short context; MTP3 at long context and four-request concurrency. Only MTP3 and MTP4 were compared at the longer lengths; neither is a universal optimum.

Uncached prefill

One request on one B65, with zero local or external cache hits.

Time is vLLM's reported request-prefill duration; rate is prompt tokens divided by that duration, not isolated kernel throughput. Short-context rows are one measured request per configuration after warmup. Long-context rows show ranges over two runs, not averages. Long-context prefill without MTP was not measured.

A fresh ~4K prompt took about 3 seconds to prefill; a fresh near-262K prompt took 13.3–13.6 minutes. These are uncached prompt-processing times, separate from decode throughput and populated-prefix cache restoration.

Context capacity

• Observed warmed KV allocation: 275,324 token slots on one B65 with the MTP3 profile. This was reported after compiled-artifact reload, not guaranteed on a cold start.
• Configured per-request limit: 262,144 tokens. The full-context workload budget was 261,616 prompt + 512 output + 16 reserve.
• Four-request workload: four 32,768-token budgets, totaling 131,072 tokens per GPU, including output allowances and reserves.
• Two-replica full-context test: simultaneous full-context requests on two independent B65s, totaling 524,288 tokens of request budgets. This is not pooled GPU memory or a 524K single-request context.

The tests also passed eight-request concurrency across two replicas, persisted hybrid attention/recurrent-state restoration, mixed local-prefix/disk restoration, and cancellation followed by recovery.

Accepted restoration outputs matched cold-reference token arrays exactly, with zero reported scheduler preemptions. Coverage combines successful checks across runs and runtime corrections, not a complete rerun on one final runtime revision.

Limits

• Runtime-specific: compatibility with unmodified upstream vLLM or other loaders is unverified.
• Not a zero-quality-loss claim: serving and cache correctness do not establish broad equivalence to the BF16 source. Upstream benchmark scores are not results for this quantization.
• Text-tested: vision tensors are retained, but image/video quality was not evaluated in this serving campaign. The multimodal task tag describes the architecture, not evaluation coverage.
• Workload-specific: the tested concurrency does not establish capacity for four maximum-length requests per GPU.

License

The pinned upstream model card identifies Apache-2.0. Retain the applicable upstream license and notices when redistributing. This independent derivative credits Qwen and documents its quantization changes above.