Qwen3.8-27B-MixedInt4-AutoRound

Qwen3.8-27B-MixedInt4-AutoRound

A mixed-precision AutoRound quantized version of Qwen/Qwen3.8-27B, optimized to reduce memory requirements while preserving the quality of the original model.

Base Model

Base model: Qwen/Qwen3.8-27B

This model is a quantized version of the original Qwen3.8-27B checkpoint. It is not a fine-tune, merge, or distillation.

Quantization

Quantization was performed using Intel AutoRound with a custom mixed-precision quantization configuration.

The quantization recipe was designed to balance:

• Model quality
• Memory efficiency
• Inference performance
• vLLM compatibility

Some model components are intentionally retained at higher precision where appropriate.

The detailed mixed-precision allocation strategy is not included in this model card.

Evaluation

Evaluation was performed using AutoRound's evaluation interface with LM Evaluation Harness.

The following results compare the original Qwen3.8-27B model against Qwen3.8-27B-MixedInt4-AutoRound.

MMLU Category Breakdown

GSM8K

Recovery Rate represents benchmark performance relative to the original Qwen3.8-27B checkpoint. A recovery rate above 100% indicates that the quantized model scored higher than the original model in that particular evaluation. Benchmark preservation does not imply identical behavior for every prompt, multimodal workload, long-context workload, or generation setting.

Usage

This checkpoint is intended for inference engines with AutoRound quantization support, including compatible versions of vLLM.

Example:

vllm serve Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --tool-call-parser qwen3_coder \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --mm-encoder-tp-mode data \
  --max-num-seqs 10 

Example with MTP

vllm serve Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --tool-call-parser qwen3_coder \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --mm-encoder-tp-mode data \
  --max-num-seqs 10 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Example with Serving 1M

vllm serve Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --tool-call-parser qwen3_coder \
  --kv-cache-dtype fp8 \
  --max-model-len 1010000 \
  --max-num-batched-tokens 8192 \
  --mm-encoder-tp-mode data \
  --max-num-seqs 10 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}'

Additional reasoning and tool-calling options should be configured according to the vLLM version being used.

Notes

• This is a quantized derivative of Qwen3.8-27B.
• The model uses a custom mixed-precision quantization recipe.
• The vision components are preserved at their original precision.
• Quantization may introduce small behavioral differences compared with the original checkpoint.

Acknowledgements

• Base model: Qwen Team
• Quantization framework: Intel AutoRound

Please refer to the original Qwen3.8-27B model card for licensing, intended usage, limitations, and other information applicable to the base model.