whisper

Whisper: Optimized for AMD ROCm

OpenAI Whisper is a transformer encoder-decoder model for automatic speech recognition (ASR) — it transcribes audio into text via autoregressive decoding. This repository packages inference for speech recognition using ONNX Runtime, exported and validated for AMD ROCm so it runs efficiently on AMD GPUs, CPUs, and NPUs.

This is based on the implementation of Whisper found here. This repository contains configurations and scripts optimized for AMD® ROCm™ platforms. You can use the whisper AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found here.

---

Task Overview

Task: Automatic speech recognition (transcription)

Dataset: Bundled 3-sample LibriSpeech smoke-test set for quick sanity checks; full-dataset accuracy uses AudioLLMs/librispeech_test_clean (2,617 samples)

Output metrics: Word Error Rate (WER), Character Error Rate (CER)

Note: The exported graph is two ONNX models — an encoder (mel spectrogram → hidden states) and an autoregressive decoder (hidden states → tokens).

---

AMD ROCm Optimization

This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points:

• Validated backends: ONNX Runtime on CPU (FP32), GPU (MIGraphX — FP32/FP16), and NPU (VitisAI — FP32 ONNX input, internally quantized).
• No code changes required versus the upstream Whisper implementation — only environment/runtime configuration differs.
• GPU pays a one-time 30+ minute MIGraphX graph-compilation cost on first run; NPU pays a similar one-time VitisAI compilation cost.

---

Getting Started

For setup instructions, evaluation scripts, and custom configuration options, see the whisper on GitHub.

---

Model Details

Model Type: Automatic speech recognition (transformer encoder-decoder)

Base Model: openai/whisper-large-v3

Model Stats:

• Exported as two ONNX graphs: encoder (mel spectrogram → hidden states) and autoregressive decoder (hidden states → tokens)
• Precision tested: FP32 (CPU, NPU); FP32, FP16 (GPU)

---

Accuracy Pipeline

Lower WER/CER means better transcription accuracy — 0% is a perfect transcription (every word/character matches the reference), while 100% means the output shares nothing with it. In practice, well-tuned Whisper models score in the low single-digit percent range on LibriSpeech test-clean; scores above ~10% indicate significant transcription errors. Scoring uses Whisper-style text normalization (transformers' BasicTextNormalizer).

Metrics Explained

Accuracy Results

Results (LibriSpeech test-clean):

<!-- accuracy-table-start --> <!-- accuracy-table-end -->

Note: These full-dataset rows are drawn from a 25-sample subset of the 2,617-sample LibriSpeech test-clean split — indicative only. A complete run over all 2,617 samples takes on the order of hours due to autoregressive greedy decoding.

---

Dig Deeper

Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?

📂 View the full project on GitHub

The GitHub repository includes:

• Setup and prerequisites for ROCm environments
• Encoder/decoder ONNX export and per-operator profiling scripts
• LibriSpeech test-clean dataset staging and WER/CER evaluation pipeline
• Benchmarking and reproduction instructions across CPU, GPU, and NPU