qwen3_omni

Qwen3-Omni: Optimized for AMD ROCm

Qwen3-Omni-30B-A3B-Instruct is a multi-modal LLM supporting text, image, audio, and video, served via vLLM's OpenAI-compatible HTTP API. This repository packages inference for multi-modal text generation using vLLM, exported and validated for AMD ROCm so it runs efficiently on AMD GPUs.

This is based on the implementation of Qwen3-Omni found here. This repository contains configurations and scripts optimized for AMD® ROCm™ platforms. You can use the qwen3_omni AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found here.

---

Task Overview

Task: Multi-modal text generation (text, image, audio, video)

Dataset: GSM8K (math, generative, 5-shot)

Output metrics: GSM8K exact-match accuracy, per-modality latency/throughput

Model variants: Default is Qwen3-Omni-30B (bf16) (Qwen/Qwen3-Omni-30B-A3B-Instruct, ~60 GB). Override with the AWQ INT4 variant (cyankiwi/Qwen3-Omni-30B-A3B-Instruct-AWQ-4bit, ~17 GB). Only one variant's weights live on disk at a time; downloading another deletes the previous one's. Both variants share Qwen3-Omni's 32768-token context ceiling.

vLLM note: GPU-only (vLLM server). No CPU or NPU inference path is available. This version of vLLM will not work with kraken2e devices — ROCm does not ship the supporting kernels for gfx1153.

---

AMD ROCm Optimization

This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm. Key points:

• Validated backend: vLLM (ROCm-enabled, built from source, OpenAI-compatible HTTP API).
• No code changes required versus the upstream Qwen3-Omni implementation — only environment/runtime configuration differs.
• The AWQ variant defaults to --dtype float16 on ROCm so vLLM selects the Exllama W4A16 kernel.
• Not supported on kraken2e devices (gfx1153) — ROCm lacks supporting kernels for this vLLM version.

---

Getting Started

For setup instructions, evaluation scripts, and custom configuration options, see the qwen3_omni on GitHub.

---

Model Details

Model Type: Multi-modal causal language model (text, image, audio, video)

Base Model: Qwen/Qwen3-Omni-30B-A3B-Instruct (default, bf16) — cyankiwi/Qwen3-Omni-30B-A3B-Instruct-AWQ-4bit (AWQ INT4) also supported

Model Stats:

• Model variant: Qwen3-Omni-30B bf16 (default, ~60 GB) — AWQ INT4 (~17 GB) also supported
• Context ceiling: 32768 tokens (shared across both variants)

---

Accuracy Pipeline

Accuracy is measured with EleutherAI's lm-evaluation-harness running GSM8K in generative mode against the server's OpenAI-compatible /v1/completions endpoint. Image, audio, and video accuracy are not covered by this text-only harness.

Metrics Explained

---

Dig Deeper

Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?

📂 View the full project on GitHub

The GitHub repository includes:

• Setup and prerequisites for ROCm environments
• Scripts for building vLLM from source, server management, and variant switching
• Per-modality (text/image/audio/video) latency and throughput benchmarking
• GSM8K accuracy evaluation and reproduction instructions