whisper-small-hybrid-int8

Whisper Small Hybrid INT8 Quantized ONNX Model

This is a quantized version of openai/whisper-small optimized for mobile deployment with ONNX Runtime.

Quantization Details

• Method: Hybrid INT8 Dynamic Quantization
• MatMul/Gemm layers: INT8 quantized (for size reduction)
• Conv layers: FP32 (excluded to avoid ConvInteger operations)
• Framework: ONNX Runtime

This hybrid approach ensures compatibility with ONNX Runtime Mobile, which does not support ConvInteger operations.

Model Specifications

Files

Usage

Python (ONNX Runtime)

import onnxruntime as ort
import numpy as np
from transformers import WhisperProcessor

# Load processor
processor = WhisperProcessor.from_pretrained("R321-hf/whisper-small-hybrid-int8")

# Load models
encoder_session = ort.InferenceSession("encoder_model.onnx")
decoder_session = ort.InferenceSession("decoder_model.onnx")

# Prepare input (mel spectrogram)
mel_spectrogram = np.random.randn(1, 80, 3000).astype(np.float32)

# Run encoder
encoder_output = encoder_session.run(None, {"input_features": mel_spectrogram})

# Run decoder with initial tokens
decoder_input_ids = np.array([[50258]])  # BOS token
logits = decoder_session.run(None, {
    "decoder_input_ids": decoder_input_ids,
    "encoder_hidden_states": encoder_output[0]
})

Android (ONNX Runtime Mobile)

// Load encoder
val encoderSession = ortEnvironment.createSession("encoder_model.onnx", sessionOptions)

// Load decoder
val decoderSession = ortEnvironment.createSession("decoder_model.onnx", sessionOptions)

// Run inference
val melTensor = OnnxTensor.createTensor(ortEnvironment, melSpectrogram, longArrayOf(1, 80, 3000))
val encoderOutput = encoderSession.run(mapOf("input_features" to melTensor))

Compatibility

• ONNX Runtime (Python)
• ONNX Runtime Mobile (Android/iOS)
• No ConvInteger operations
• NNAPI compatible

License

MIT License (same as original Whisper model)

Credits

• Original model: openai/whisper-small
• ONNX export: onnx-community/whisper-small
• Quantization: Hybrid INT8 with ONNX Runtime