Qwen3-ASR-1.7B-nvfp4

Qwen3-ASR-1.7B NVFP4 Quantized

NVFP4 quantized version of Qwen3-ASR-1.7B using NVIDIA ModelOpt (Max calibration algorithm), ready for deployment with vLLM.

Quantization Details

Only the LLM component (model, ~1.4B params, 82% of total) is quantized to NVFP4. audio_tower and lm_head remain in BF16.

Calibration algorithm: Max (per-tensor symmetric, MaxCalibrator)

Results

Memory Usage (RTX 5090, gpu_memory_utilization=0.7)

WER (760 VIVOS test samples, concurrency=1)

Throughput across Concurrency Levels (7168 VIVOS samples)

NVFP4 achieves highest throughput at high concurrency due to smaller model size freeing more KV cache capacity.

LLM Output Cosine Similarity vs BF16 (50 samples)

Usage

Serving with vLLM

git clone https://github.com/QwenLM/Qwen3-ASR
pip install nvidia-modelopt vllm

qwen-asr-serve vrfai/qwen3asr-nvfp4 \
    --quantization modelopt_fp4 \
    --gpu-memory-utilization 0.7

Inference

import requests, soundfile as sf, io

def transcribe(audio_path, url="http://localhost:8000/v1/audio/transcriptions"):
    audio, sr = sf.read(audio_path)
    buf = io.BytesIO()
    sf.write(buf, audio, sr, format="WAV")
    buf.seek(0)
    r = requests.post(
        url,
        files={"file": ("audio.wav", buf, "audio/wav")},
        data={"model": "vrfai/qwen3asr-nvfp4"},
    )
    return r.json().get("text", "")

References

• Qwen3-ASR Technical Report
• NVIDIA ModelOpt
• Quantization code

Quantization Script

The recipes and scripts used to quantize this model can be found in the following repository:

• VinRobotics/model-quantization-recipes