Qwen3-4B-FP8

Qwen3-4B-FP8

FP8 (W8A8 dynamic) quantization of Qwen/Qwen3-4B, produced with llm-compressor in the compressed-tensors format for efficient serving with vLLM and SGLang.

8-bit floating point weights with dynamic activation scales; roughly half the size of bf16, near lossless, and native to Hopper/Blackwell GPUs.

Serving / 部署

vllm serve qtum/Qwen3-4B-FP8

The quantization scheme is declared in config.json and detected automatically — no extra flags are required. Compatible with any engine that reads compressed-tensors (vLLM, SGLang, …).

config.json 中已声明量化方案，引擎自动识别，无需额外参数。兼容一切支持 compressed-tensors 的推理框架。

Prompt format / 对话格式

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Details / 说明

English — This checkpoint reproduces the base model's behavior at reduced precision, shrinking memory footprint and increasing throughput while keeping quality close to the original. It is a drop-in replacement for the base model in vLLM/SGLang deployments.

中文 — 本权重是原始模型的低精度量化版本，在保持接近原模型质量的同时，显著降低显存占用、提升吞吐。 在 vLLM/SGLang 部署中可直接替换原始模型使用。

---

Quantized by qtum. Weights are unmodified apart from quantization; all model behavior and license obligations follow the base model. / 由 qtum 量化，除量化外未改动权重，模型行为与许可义务均遵循原始模型。