Downloads · 30 days
46
36% of all-time downloads
qwertz92/granite-speech-5.0-470m-turboctc-onnx
granite-speech-5.0-470m-turboctc-onnx is a automatic speech recognition model from qwertz92. Use it when you need speech turned into text. It is set up for onnx. The card lists the license as apache-2.0.
Reproducible FP32, FP16, and dynamic INT8 ONNX exports of IBM's English-only granite-speech-5.0-470m-turboctc. The repository includes conversion scripts, exact source/output hashes, independent PyTorch comparisons, a…
Downloads · 30 days
46
36% of all-time downloads
All-time downloads
128
Public
Repo size
3.4 GB
Likes
0
Public
Click a slice to open those files.
.onnx3.4 GB · 100%
From the Hugging Face model README
Reproducible FP32, FP16, and dynamic INT8 ONNX exports of IBM's English-only
granite-speech-5.0-470m-turboctc.
The repository includes conversion scripts, exact source/output hashes,
independent PyTorch comparisons, and local CPU/DirectML measurements.
This is a community conversion, not an IBM release. The graphs are not yet
integrated into stt_app.
| File | Precision | Bytes | Size | SHA-256 | Recommended use |
|---|---|---|---|---|---|
onnx/model.onnx | FP32 | 1,892,339,420 | 1.763 GiB | cc647a0681142d6651dc14bf582cf2ef06da1b1d0978f078bf66b56a407fcddb | Best measured DirectML speed on Intel Arc A750 |
onnx/model_fp16.onnx | FP16 weights/compute, FP32 I/O | 946,797,887 | 902.937 MiB | 5ae14135ac9ac0fbdce8ab8566dc45de5d25111baa0db43d94f4821d7b284359 | Half-size GPU-compatible alternative; avoid CPU |
onnx/model_int8.onnx | Dynamic per-channel QInt8 MatMul weights | 551,294,349 | 525.756 MiB | 8173a50ea67f864801971a40622a2e5c7d62fd230912bde94b745f92c74d60e9 | Fastest and smallest tested CPU variant |
INT8 is a mixed graph, not an all-INT8 network. It has 163
MatMulInteger nodes and 326 INT8 initializers. Another 49 MatMul nodes,
convolutions, normalization, softmax, and other operations remain floating
point. Activations entering quantized matrix multiplications are quantized at
runtime.
The upstream Safetensors checkpoint contains 502 BF16, 32 FP32, and 16 INT64 tensors. Most FP32 ONNX weights are therefore exact FP32 expansions of BF16 source values, not newly gained precision. FP16 changes floating-point storage and execution precision. INT8 is genuinely lossy quantization.
input_features, float32 [batch_size, feature_frames, 320]logits, float32
[batch_size, floor(feature_frames / 4), 16384]Audio feature extraction and greedy CTC decoding stay outside the graph. Use
the bundled processor configuration with transformers>=5.16.0. Equal-length
clips can be batched. This first graph has no attention-mask input, so run
unequal-length clips separately instead of padding them into one batch.
import numpy as np
import onnxruntime as ort
import soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor
repo_id = "qwertz92/granite-speech-5.0-470m-turboctc-onnx"
processor = AutoProcessor.from_pretrained(repo_id)
model_path = hf_hub_download(repo_id, "onnx/model_int8.onnx")
audio, sample_rate = sf.read("audio.wav", dtype="float32")
inputs = processor(audio, sampling_rate=sample_rate, return_tensors="np")
session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
logits = session.run(
["logits"],
{"input_features": np.asarray(inputs.input_features, dtype=np.float32)},
)[0]
token_ids = logits.argmax(axis=-1)
print(processor.batch_decode(token_ids)[0])
The reference was generated independently with PyTorch 2.11 from the pinned IBM checkpoint. The suite covers 20 pinned LibriSpeech clips, ten synthetic lengths around subsampling/block boundaries, and a batch-size-two case.
| Variant | Exact CTC token streams | Exact decoded transcripts | Notes |
|---|---|---|---|
| FP32 | 31/31 | 31/31 | Max absolute logit difference 0.0009813 |
| FP16 | 31/31 | 31/31 | Same argmax tokens; expected floating-point logit differences |
| Dynamic INT8 | 30/31 | 30/31 | 19/20 exact real-audio transcripts; minimum frame argmax agreement 98.53% |
The one INT8 transcript difference occurred on the longest 29.4-second clip.
FP32 produced and a paintings, INT8 produced and adam paintings, and the
dataset reference reads AND AT EM PAINTINGS. This is useful graph-health
evidence, but 20 clips are not enough to claim equal WER or that INT8 is more
accurate.
The lower-precision JSON comparisons intentionally show "passed": false:
the validator's pass threshold requires near-FP32 numerical parity. Both graphs
passed the full ONNX checker and the functional results above; the reports do
not relabel lossy conversion as numerically identical.
One 29.4-second, mono 16 kHz LibriSpeech clip was measured after one warm-up. The table reports median warm graph inference over five runs, excluding the separately measured 0.11-0.12-second feature extraction step.
onnxruntime-node 1.24.3| Variant | CPU load | CPU inference | CPU RTFx | DirectML load | DirectML inference | DirectML RTFx |
|---|---|---|---|---|---|---|
| FP32 | 9.64 s | 0.968 s | 30.36x | 2.30 s | 0.107 s | 275.55x |
| FP16 | 4.99 s | 8.495 s | 3.46x | 1.32 s | 0.179 s | 164.39x |
| Dynamic INT8 | 3.03 s | 0.282 s | 104.43x | 1.50 s | 0.137 s | 214.08x |
FP16 is slow on the tested CPU because the CPU execution provider lacks native kernels for relevant FP16 operations. On this DirectML/Arc combination, FP32 was faster than both reduced-precision graphs. Smaller precision is therefore not automatically faster; choose by target runtime and measured hardware.
CPU scaling also rules out a simple "memory-bound" label. From one to six threads, FP32 sped up 2.84x and INT8 3.23x. From six to twelve threads, gains fell to 1.45x and 1.25x. Warm inference uses all cores and reuses weights across many audio frames, so compute, memory traffic, and runtime overhead all matter. Cold loading tracks graph size more directly: INT8 reduced load from 9.64 to 3.03 seconds and peak process RSS from about 3.72 GB to 1.71 GB. No hardware performance counters were available, so this does not claim a measured memory bandwidth ceiling.
18ca3c1de6cd092b5a30c39fb0f04550b38ed1a08b98a8c34fd5fcb081caef719638eded31bb6d197d62053eefc5c1703aaf1ad46467bcfreports/conversion/The export changes only the two temporal-subsampling residual unfold
operations to mathematically equivalent avg_pool1d operations so dynamic time
lengths survive legacy ONNX export. A full-model pre-export check measured a
maximum difference of exactly 0.0 before serialization. Every published graph
passed onnx.checker.check_model(..., full_check=True).
Apache-2.0, following the upstream IBM model. See the upstream model card for training data, intended use, limitations, and IBM's citation.