Qwen3-ASR-1.7B-fp8

Qwen3-ASR-1.7B FP8 Quantized

FP8 quantized version of Qwen3-ASR-1.7B using NVIDIA ModelOpt (Max calibration algorithm), ready for deployment with vLLM.

Quantization Details

Only the LLM component (model, ~1.4B params, 82% of total) is quantized to FP8. audio_tower and lm_head remain in BF16.

Calibration algorithm: Max (per-tensor symmetric, MaxCalibrator)

Results

Memory Usage (RTX 5090, gpu_memory_utilization=0.7)

WER (760 VIVOS test samples, concurrency=1)

Throughput across Concurrency Levels (7168 VIVOS samples)

LLM Output Cosine Similarity vs BF16 (50 samples)

Usage

Serving with vLLM

git clone https://github.com/QwenLM/Qwen3-ASR
pip install nvidia-modelopt vllm

qwen-asr-serve vrfai/qwen3asr-fp8 \
    --quantization modelopt \
    --gpu-memory-utilization 0.7

Inference

import requests, soundfile as sf, io

def transcribe(audio_path, url="http://localhost:8000/v1/audio/transcriptions"):
    audio, sr = sf.read(audio_path)
    buf = io.BytesIO()
    sf.write(buf, audio, sr, format="WAV")
    buf.seek(0)
    r = requests.post(
        url,
        files={"file": ("audio.wav", buf, "audio/wav")},
        data={"model": "vrfai/qwen3asr-fp8"},
    )
    return r.json().get("text", "")

References

• Qwen3-ASR Technical Report
• NVIDIA ModelOpt
• Quantization code

Quantization Script

The recipes and scripts used to quantize this model can be found in the following repository:

• VinRobotics/model-quantization-recipes