smoothquant-qwen25-15b-g52xlarge

SmoothQuant Qwen/Qwen2.5-1.5B-Instruct

Quantized derivative of Qwen/Qwen2.5-1.5B-Instruct, made with QuantPort. Not Hub fp16. Weights are the packed/runtime artifact from job 20260920T220639Z-9e4555.

WikiText-2 (test, 2048-token windows, 65504 tokens)

Measured on NVIDIA A10G (g5.2xlarge) against the original fp16 snapshot. Numbers copied from jobs/20260920T220639Z-9e4555/benchmark.json.

• ppl_ratio: 1.050919 (quality_ok=True)
• improved_throughput: False
• improved_vram: True

Pull

from huggingface_hub import snapshot_download
path = snapshot_download("chris320211/smoothquant-qwen25-15b-g52xlarge")

This is not a drop-in AutoModelForCausalLM.from_pretrained checkpoint. Reload with the included quant_agent_inference_adapter.py after the method repo (and overlay, if any) is on QUANT_AGENT_METHOD_REPO. See quantization_config.json.

<!-- quant-agent:vllm -->

Serve with vLLM

A native vLLM build of these weights: chris320211/smoothquant-qwen25-15b-g52xlarge-vllm.

vllm serve chris320211/smoothquant-qwen25-15b-g52xlarge-vllm

vLLM PPL ratio 1.0148 against vLLM fp16; prefill 1.85×, decode 1.25× (batch 1), 1.42× (batch 16). Measured on NVIDIA GeForce RTX 3090 with vLLM 0.30.0.

vLLM results

Measured on NVIDIA GeForce RTX 3090: the vLLM build against vLLM fp16, and this repo's weights in plain Transformers against the original model.

• Transformers columns: the native artifact chris320211/smoothquant-qwen25-15b-g52xlarge through its inference adapter, and Qwen/Qwen2.5-1.5B-Instruct in fp16, in Transformers 4.46.3 (torch 2.7.1+cu128) on the same GPU. Transformers prefills 8 windows per forward; vLLM schedules all 32.
• Against Transformers fp16: vLLM int8 prefill 2.04×, decode 18.28× at batch 1, 16.68× at batch 16. The native artifact in Transformers: prefill 1.19×, decode 0.41× at batch 1, 0.39× at batch 16.

For serving, use chris320211/smoothquant-qwen25-15b-g52xlarge-vllm. In plain Transformers this artifact decodes slower than fp16; its gain there is memory.

<!-- /quant-agent:vllm -->

License

Inherits the license of Qwen/Qwen2.5-1.5B-Instruct (apache-2.0), plus the method repository license. Keep LICENSE / NOTICE.md from the snapshot when present.