Qwen3-VL-4B-Instruct-SmoothQuant-W8A8-FP8

Qwen3-VL-4B-Instruct-SmoothQuant-W8A8-FP8

This repository provides a SmoothQuant-based compressed version of Qwen3-VL-4B-Instruct for efficient multimodal inference and evaluation.

Overview

This model is a third-party compressed checkpoint built on top of Qwen3-VL-4B-Instruct, mainly for efficient deployment, benchmarking, and PTQ baseline construction.

The current release follows a SmoothQuant-style W8A8-FP8 workflow in the llm-compressor framework. In this project, the checkpoint is released as a deployment-oriented baseline for FP8-compatible serving and evaluation.

Compared with the original checkpoint layout, this release also reduces storage footprint in a practical way.

• Original size: 4,850,810 KB + 3,816,885 KB
• Quantized size: 4,873,795 KB + 245,804 KB
• Compression: -40.935%

Base Model

• Base model: Qwen/Qwen3-VL-4B-Instruct
• Model family: Qwen3-VL
• Quantization method: SmoothQuant
• Quantization format: W8A8-FP8
• Framework: llm-compressor

Quantization Setup

This release follows a SmoothQuant-based post-training workflow in llm-compressor, where activation outliers are first smoothed and the model is then exported through an FP8-oriented low-precision deployment path.

Quantization Recipe

recipe = [
    SmoothQuantModifier(
        smoothing_strength=0.5,
        mappings=SMOOTHQUANT_MAPPINGS,
        ignore=["re:.*visual.*"],
        num_calibration_steps=NUM_CALIBRATION_SAMPLES,
    ),
    QuantizationModifier(
        targets="Linear",
        scheme="FP8",
        ignore=["re:.*lm_head", "re:.*visual.*"],
    ),
]

Notes

• SmoothQuantModifier is used to smooth activation outliers before quantization.
• smoothing_strength=0.5 balances the migration of quantization difficulty between activations and weights.
• mappings=SMOOTHQUANT_MAPPINGS defines how the smoothing transform is applied across eligible modules.
• num_calibration_steps=NUM_CALIBRATION_SAMPLES matches the calibration subset used during PTQ.
• After smoothing, QuantizationModifier(..., scheme="FP8") is used to quantize target Linear layers for deployment-oriented inference.
• lm_head and visual modules are excluded from this quantization path in the released checkpoint.

Calibration Setup

Calibration data was constructed from the Flickr30k image-caption dataset.

For SmoothQuant calibration, 128 samples were selected from local Flickr30k parquet files after dataset loading and random shuffling with a fixed seed (seed=42). Each sample was converted into a multimodal chat-style input containing one image and one paired caption, and then processed into model inputs such as input_ids, attention_mask, pixel_values, and image_grid_thw.

Calibration Details

• Dataset: Flickr30k
• Data format: local parquet files
• Number of calibration samples: 128
• Sampling strategy: shuffled subset with fixed random seed
• Max sequence length: 2048
• Purpose: multimodal activation calibration for SmoothQuant-based PTQ

Evaluation Configuration

For evaluation in VLMEvalKit, the following model entry can be added to VLMEvalKit/vlmeval/config.py:

'Qwen3-VL-4B-Instruct-SmoothQuant-W8A8-FP8': partial(
    vlm.Qwen3VLChat,
    model_path='/home/lml/models/Qwen3-VL-4B-Instruct-SmoothQuant-W8A8-FP8',
    min_pixels=256 * 28 * 28,
    max_pixels=1280 * 28 * 28,
    use_custom_prompt=False,
    use_vllm=True,
    temperature=0.7,
    max_new_tokens=8192,
    repetition_penalty=1.0,
    presence_penalty=1.5,
    top_p=0.8,
    top_k=20,
    max_model_len=16384,
    gpu_utils=0.90,
    enable_thinking=False,
)

Intended Use

This release is intended for:

• Efficient multimodal inference
• PTQ baseline construction for Qwen3-VL
• Evaluation with VLMEvalKit
• Serving experiments with vLLM
• Research on VLM post-training quantization and deployment-oriented low-precision inference

Disclaimer

This is a third-party quantized checkpoint and is not an official release from the Qwen team.

The checkpoint is provided as a practical SmoothQuant-style baseline. Actual runtime behavior and performance can depend on the serving backend and hardware support path.

Citation

If you use this model, please cite the original Qwen3-VL report, SmoothQuant, VLMEvalKit, and the calibration dataset when appropriate.

@article{bai2025qwen3vl,
  title={Qwen3-VL Technical Report},
  author={Bai, Shuai and Cai, Yuxuan and Zhu, Keming and others},
  journal={arXiv preprint arXiv:2511.21631},
  year={2025}
}

@article{xiao2023smoothquant,
  title={SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models},
  author={Xiao, Guangxuan and Lin, Ji and Sehwag, Vikhram and others},
  journal={Proceedings of the International Conference on Machine Learning (ICML)},
  year={2023}
}

@misc{duan2024vlmevalkit,
  title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Vision-Language Models},
  author={OpenCompass Team},
  howpublished={\url{https://github.com/open-compass/VLMEvalKit}},
  year={2024}
}

@article{young2014image,
  title={From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions},
  author={Young, Peter and Lai, Alice and Hodosh, Micah and Hockenmaier, Julia},
  journal={Transactions of the Association for Computational Linguistics},
  volume={2},
  pages={67--78},
  year={2014},
  publisher={MIT Press}
}

Acknowledgement

This repository builds upon the following open-source projects:

• Qwen3-VL
• llm-compressor
• AWQ
• VLMEvalKit
• vLLM