Qwen3-ASR-0.6B-ONNX-CPU

Qwen3-ASR-0.6B — Full ONNX CPU

Run Qwen3-ASR on any CPU. No GPU. No PyTorch.

Self-contained ONNX pipeline for Qwen3-ASR-0.6B. Encoder and decoder both run on ONNX Runtime with INT8 quantized decoder. Long audio is automatically split at silence boundaries — no manual chunking needed.

Highlights

• Zero GPU, Zero PyTorch — onnxruntime + librosa + tokenizers only
• Real-time on 8W CPU — RTF 0.71x with VAD chunking on Intel N100
• 3x realtime on desktop — RTF 0.32x (INT8)
• Self-contained — all weights + tokenizer included
• 30 languages — same multilingual coverage as the original
• Long audio support — automatic silence-based splitting with --chunk-sec
• Bit-exact encoder — cosine similarity 1.000000 vs PyTorch reference

Quick Start

git clone https://huggingface.co/Daumee/Qwen3-ASR-0.6B-ONNX-CPU
cd Qwen3-ASR-0.6B-ONNX-CPU

python3 -m venv .venv && source .venv/bin/activate
pip install onnxruntime librosa soundfile tokenizers

# Short audio
python onnx_inference.py test_audio/librispeech_1_1089_1.wav

# Long audio (auto-chunked at silence)
python onnx_inference.py long_meeting.wav --chunk-sec 30

# Specify language + JSON output
python onnx_inference.py audio.wav --language Korean --json

Benchmarks

Intel N100 (4 cores, 8W TDP)

Short audio — 13 LibriSpeech test-clean samples, INT8 decoder:

Audio > 10s achieves realtime or faster. Decoder: ~100ms/token (INT8).

Long audio — production deployment with VAD chunking (Docker, 2 threads):

Without chunking, 10-min audio consumes 15GB+ and gets OOM-killed. With 30s chunks, prefill cost is amortized → RTF drops well below 1.0x.

Desktop x86_64:

Architecture

Long Audio

Audio longer than 45s is automatically split at silence boundaries using RMS energy detection. No external VAD model needed.

python onnx_inference.py meeting.wav                  # 30s chunks (default)
python onnx_inference.py meeting.wav --chunk-sec 20   # 20s chunks, less memory

Split range scales with target: min = target/2, max = target×1.5. The split point is the silence frame nearest to the target length.

Files

Total: ~2.5 GB

Model

30 languages: Chinese, English, Cantonese, Japanese, Korean, Arabic, German, French, Spanish, Portuguese, Indonesian, Italian, Russian, Thai, Vietnamese, Turkish, Hindi, Malay, Dutch, Swedish, Danish, Finnish, Polish, Czech, Filipino, Persian, Greek, Romanian, Hungarian, Macedonian

Technical Notes

• Attention: Original cu_seqlens windowed attention only works with flash_attention_2. CPU uses eager mode (all-to-all). ONNX export matches this.
• Weight tying: embed_tokens = lm_head in original. In ONNX, separated — embed_tokens.bin for input, lm_head baked into decoder.
• KV Cache: [num_layers, batch, kv_heads, seq_len, head_dim] — init outputs, step extends.
• MRoPE: Layout [24,20,20]. For ASR (no vision), all 3 dims share the same position IDs.

Dependencies

onnxruntime
librosa
soundfile
tokenizers

No PyTorch. No transformers. No CUDA.

Acknowledgements

Decoder ONNX export architecture inspired by andrewleech/qwen3-asr-onnx.

References

• Qwen3-ASR-0.6B — Original model
• Qwen3-ASR GitHub — Official repo
• Technical Report — arXiv:2601.21337
• andrewleech/qwen3-asr-onnx — ONNX export reference

License

Code: Apache 2.0. Model weights: original license.