Qwen2.5-Coder-7B-AutoRound-W4A16

Qwen2.5-Coder-7B AutoRound W4A16

This repository contains a quality-oriented W4A16 quantization of Qwen/Qwen2.5-Coder-7B, intended primarily for low-latency code completion and fill-in-the-middle (FIM) inference.

This is the base model, not the instruction-tuned variant. It is most useful through the completions API with Qwen's FIM tokens.

Model details

• Base model: Qwen/Qwen2.5-Coder-7B
• Exact base revision: 0396a76181e127dfc13e5c5ec48a8cee09938b02
• Quantizer: AutoRound 0.15.0
• Weight format: INT4 / W4A16, symmetric, group size 128
• Runtime activations: BF16
• Quantized linear layers: 196 of 197
• lm_head: BF16
• Packing format: auto_round:auto_gptq
• Checkpoint size: approximately 5.2 GiB
• Measured model-weight VRAM: 5.17 GiB
• Tested inference path: vLLM 0.27.1, automatic inc detection, MarlinLinearKernel

No re-quantization is required for NVIDIA Ampere GPUs. The intended target is an RTX A6000 48 GB, although the published reference measurements below were collected on an A100 PCIe 80 GB.

Quantization

The model was quantized with 1,000 AutoRound optimization iterations per layer and 512 decontaminated calibration sequences of 2,048 tokens each. The calibration mixture was:

• 50% FIM code
• 25% plain code
• 25% coding instruction plus code

The source dataset was nvidia/OpenCodeInstruct at revision 8f3ba5bafe4d6e8db46082cf7ae6741bc370604d. A 13-gram overlap check against HumanEval rejected 479 of 4,141 examined source records before calibration packing.

Quantization took 3,535.86 seconds on one A100 80 GB and peaked at 24.72 GB of VRAM. This is quantization memory, not inference memory.

Evaluation

A deterministic custom HumanEval FIM test preserved the first and final third of each canonical solution, generated the missing middle, reconstructed the solution, and executed all 164 tasks in a restricted container.

Pairwise, W4A16 lost four tasks solved by BF16 and gained nine tasks not solved by BF16. The aggregate result should therefore not be interpreted as proof that quantization generally improves model quality. This is a custom FIM evaluation, not the standard HumanEval generation protocol.

On an A100 PCIe 80 GB, using an uncached 1,024-token prompt and producing 64 tokens:

These absolute numbers are not predictions for the RTX A6000. Re-run latency acceptance tests on the deployment GPU.

vLLM usage

The following is the recommended starting point for one to three small concurrent requests on an RTX A6000 48 GB:

vllm serve issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 \
  --served-model-name autocomplete \
  --generation-config vllm \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --gpu-memory-utilization 0.25 \
  --max-model-len 8192 \
  --max-num-seqs 3 \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --performance-mode interactivity \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}'

The quantization mode should be detected automatically. N-gram speculative decoding is deliberately not recommended for the intended mix of small requests because it increased TTFT and tail latency in the reference tests.

Fill-in-the-middle prompt

prompt = (
    "<|fim_prefix|>" + prefix
    + "<|fim_suffix|>" + suffix
    + "<|fim_middle|>"
)

Send the prompt to a text-completions endpoint rather than treating the base model as a chat model.

Limitations

• Quantization can change individual outputs even when aggregate benchmark quality is retained.
• The model is not instruction tuned and is not a drop-in replacement for a chat assistant.
• Only an 8,192-token runtime configuration was benchmarked for this checkpoint.
• Generated code should be reviewed and executed in an appropriately isolated environment.
• This model inherits the limitations and license conditions of the upstream Qwen model.

License and attribution

The checkpoint is a quantized derivative of Qwen/Qwen2.5-Coder-7B and is distributed under the Apache License 2.0. See LICENSE and the upstream model card for details.