Downloads · 30 days
227
42% of all-time downloads
musetric/whisper-large-v3-turbo-onnx
whisper-large-v3-turbo-onnx is a automatic speech recognition model from musetric. Use it when you need speech turned into text. It is set up for transformers.js. The card lists the license as mit.
ONNX export of [openai/whisper-large-v3-turbo][base] in the [transformers.js][tjs] layout, with the cross-attention alignment heads baked into generationconfig.json so the decoder emits word-level timestamps (returnti…
Downloads · 30 days
227
42% of all-time downloads
All-time downloads
545
Public
Repo size
4.3 GB
Likes
0
Public
Click a slice to open those files.
.onnx_data776 MB · 99%
From the Hugging Face model README
ONNX export of openai/whisper-large-v3-turbo in the
transformers.js layout, with the cross-attention alignment heads baked
into generation_config.json so the decoder emits word-level timestamps
(return_timestamps: 'word'). Packaged for the musetric
packages/ai runtime (@huggingface/transformers on the WebGPU
execution provider).
Turbo keeps a same-shaped encoder and replaces the 32-layer decoder with a
4-layer one, so the merged decoder graph is a quarter the size of large-v3's. It
is produced by scripts/onnx/whisper in musetric/lab
(optimum + the transformers.js converter).
| file | role |
|---|---|
encoder_model_q4.onnx | audio encoder, q4, 433 MB with its weights in encoder_model_q4.onnx_data — attention split into query blocks, and the stem written without Conv or a 2-axis Transpose, so it both binds and computes correctly on mobile GPUs, see below |
decoder_model_merged_fp16.onnx | decoder step, float16, 318 MB with its weights in decoder_model_merged_fp16.onnx_data; the cached step alone, without the If of a merged decoder, so it needs cross_kv_fp16.onnx and a host that feeds the prompt one token at a time, see below; attention scores kept off the kernel Adreno 600 breaks, and its small products kept out of the float16 subnormal range Adreno computes as zero |
cross_kv_fp16.onnx | the encoder states' key and value projections of the four decoder layers, float16, 26 MB with its weights in cross_kv_fp16.onnx_data — run once per window before the step |
config.json, generation_config.json, preprocessor_config.json | model / feature-extractor config |
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, normalizer.json | tokenizer |
Each graph keeps its weights as ONNX external data in <graph>_data, every
tensor equal to the one revision c6114a37 held inside the graph. A browser
host hands that file to onnxruntime-web as externalData, and the WebGPU
provider uploads the weights from it to the GPU; weights inside a graph pass
through the wasm heap first, and the heap never shrinks. With these files and
the three sessions created one after another, the musetric runtime keeps
272 MiB of wasm memory where it kept 1080 to 1445 MiB, with the same output
(musetric/musetric#1088).
The encoder always sees a 30-second window — 1500 frames, 20 heads, 64 per head
— and a plain export computes each layer's attention in one shot, so its score
tensor is [20, 1500, 1500] float32: 171.7 MiB, in every one of the 32
layers. WebGPU guarantees only 128 MiB per storage binding, and current mobile
GPUs report exactly that minimum, so such a graph cannot run there at all —
onnxruntime fails the Softmax dispatch that binds it.
Softmax normalizes each query row over the full key axis independently, so
splitting the queries into blocks and concatenating the per-block outputs is
exact, not an approximation and not the online-softmax bookkeeping flash
attention needs. block_attention.py in the export tooling rewrites each
MatMul → Softmax → MatMul chain into one chain per 250-row block plus a
Concat: peak score tensor 28.6 MiB, graph 1751 → 2423 nodes, file 80 KB
larger. No weight is read, let alone requantized — the q4 MatMulNBits nodes are
copied through — and the CPU provider produces bit-identical output either way.
On a GPU with room for the unblocked graph the rewrite costs about 1 %.
Conv and no 2-axis TransposeAdreno 600-series adapters compute one Transpose shape wrong: a quarter of the
output comes back as zeros. It is not a precision loss — a permutation cannot
change values, yet the output RMS drops by exactly the √0.75 that a zeroed
quarter implies — and nothing reports a fault: no NaN, no error, no fallback.
The cause is in the execution provider's shader rather than in the export. It
stages the transpose through a workgroup tile declared as
array<array<T, tile_size + 1>, tile_size> — 16 × 17 float32, 1088 bytes —
where the + 1 is padding against bank conflicts. On that hardware a workgroup
array whose size is not a multiple of 512 bytes loses the stores of one warp
out of four across workgroupBarrier: the stores land, and a thread reads its
own slot back correctly, but the other threads never see them. Widening the
padded row so the tile lands on 1536 bytes fixes it upstream in one line.
Only the shapes that reach that shader are affected: once leading extents of one
are squeezed away, a rank-2 permutation swapping both axes. That is one node in
this graph — plus the two Conv nodes, because the provider transposes NCHW to
NHWC inside them. The permutations attention uses (rank 4, and rank 3 with a
real batch) take a different kernel and are exact.
Both are gone from this export, with the arithmetic unchanged:
Pad → Slice ×3 → Concat → Reshape → MatMul → Add → Reshape,
the same convolution written with the weight on the left so that nothing has
to be turned around;Gather against a precomputed index
table, which costs 7.3 MiB of int32 and returns the identical permutation.Graph 2423 → 2441 nodes, and against the CPU provider an affected adapter goes
from visibly wrong output to the same agreement a healthy one reports. The
rewrites are conv_to_matmul.py and transpose_to_gather.py in the
export tooling, applied together by mobile_encoder.py.
Because the index table is static, the encoder is now fixed at batch 1. The runtime feeds one 30-second window at a time, which is what it already did.
On this decoder q4 buys almost nothing. Only 52 MB of its 334 MB is quantized weight — the rest is the embedding, which stays float32 either way — so 4-bit saves 10 MB while paying full quantization error across all four decoder layers. With 32 layers that error washes out; with four it does not. The encoder is a different case and stays q4.
Switching the decoder from q4 to float16 costs 10 MB of download and lowers peak VRAM from 4089 MiB to 3101 MiB, because the KV cache becomes float16 as well. End-to-end speed is unchanged: the decode step is faster, but model load and the encoder dominate the wall clock.
Accuracy is not uniformly better in a way a word-error rate will show — the average barely moves. What it removes is a failure mode: on repetitive non-English material the q4 decoder sometimes collapses instead of transcribing, and float16 does not. English output is unchanged.
The graph is float16 throughout except at three boundaries:
encoder_hidden_states stays float32 so the q4 encoder can feed it directly,
logits is cast back to float32, and cross_attentions.* come out of a
float32 softmax (see the scaling section below), because the word-timestamp DTW
reorders words within a line when run on float16 attention maps. The KV cache
is float16 end to end.
The WebGPU execution provider multiplies A [batch, M, K] × B [batch, K, N] with
a packed vec4 shader whenever both K and N are multiples of four. On Adreno
600-series adapters that shader returns wrong values when M is small — and a
decoder has one query row per cached step and the prompt on the first. With the
head size K = 64 and N the key length, every N that is a multiple of four
but not of 32 comes back as garbage, in float16 and float32 alike, with no NaN
and no error. Self-attention reaches such a length on every fourth step as the
cache grows; cross-attention has 1500 keys and reaches it on every step. The
corrupted hidden state then feeds the next layers' keys and values, so the cache
carries the error to every later step.
Each attention score MatMul(Q, Transpose(K)) now carries one zero column on
the query and one zero row on the transposed key, so the product runs with an
inner extent of 65 and the provider takes its scalar kernel; the added row and
column multiply to zero, the output keeps its shape, and on the CPU provider the
float16 graph returns the same logits bit for bit. The head size is known at
export, so unlike a pad of the key length this needs no shape arithmetic at run
time, which ran on the CPU provider and split every step around each attention:
one token of a 30 s window went from 64.1 to 37.0 ms on a desktop NVIDIA and
from 324.9 to 266.3 ms on an Adreno 660. The products are rewritten by
mobile_decoder.py in the export tooling.
On Adreno 660 the stock decoder drifted up to 7.1 in the logits from the wasm
provider on the same device, where a desktop NVIDIA adapter stays within 0.4, and
in the musetric app the end of a line came out different from desktop. With the
rewrite the drift is 1.6, the greedy tokens match wasm and the app transcribes
the same text as on desktop.
Three rewrites take the decoder off what a generation step does not need,
measured on the musetric runtime with 30 s windows (revision 81da5251 before
them):
trim_cross_attentions.py narrows
each cross_attentions.* output to the heads alignment_heads names and
rewrites the config to the new positions: one, one, two and four heads per
layer. Bit-identical on the CPU provider; on an Adreno 660 a token goes from
197.2 to 174.0 ms.step_outputs carries the logits and the kept heads
flattened into one tensor; the original outputs stay. Fetching it and the
cache alone takes a step on a desktop NVIDIA from 31.3 ms with five tensors to
18.6 ms with one.If. A merged decoder keeps its first and its cached step as the two
branches of an If, and ONNX Runtime fuses nothing through it: a cached step
runs as some 650 small kernels, 20.1 ms on a desktop NVIDIA and 49.7 ms on an
Adreno 750 with the outputs kept on the device, against 10.4 and 30.9 ms for
the same branch as a flat graph. decoder_model_merged_fp16.onnx is that
cached branch alone, with encoder_hidden_states kept as an input nothing
reads; cross_kv_fp16.onnx holds the part of the first branch it lacks, the
encoder states' key and value projections of every layer. The first step is
the projection plus the prompt fed through the step one token at a time, which
is exactly the causal mask the first branch applies to the whole prompt. On
the CPU provider that matches the merged first step to 0.004 at a logit scale
of 21 and a cached step to 0.006, float16 rounding, with the same argmax.The step has no use_cache_branch input, which is how the musetric runtime
tells it from a merged decoder. Eight 30 s windows of the runtime, four per
call, take 25 % less on a desktop NVIDIA and 14 % less on an Adreno 660 than
with 81da5251, with the same words and the same word times bit for bit; on an
Adreno 750 the encoder holds the batch and the difference stays inside the heat
spread. The alignment pass of a window takes 32 % less on the desktop and 22 %
less on both phones.
Adreno GPUs compute float16 without subnormals. A product below 2^-14 is zero
there even when both factors are normal numbers - 2^-7 times 2^-8 is zero -
and a stored value below 2^-14 reads as zero; an NVIDIA GPU keeps both. Three
products of a decoder step are sums of exactly such terms: the attention
probabilities times the values, the attention output times out_proj, and
the GELU tail times fc2. With revision 4d1b2589 an Adreno 750 lost 44 %,
20 % and 34 % of the scale of those results at layer 0 against about 1 % on a
desktop NVIDIA, and a float16 softmax dropped the cross-attention probabilities
below 2^-14 on top: 1-2 % of a head's mass, up to a fifth of the layer 0
context.
Forced to the same tokens, the step's logits were up to 2.0 from the CPU
provider on both Adreno GPUs against 0.4 on the desktop, and a phone returned
other punctuation and other word times than the desktop.
scale_residual_stream.py in the export tooling scales what those
products read by 2^10, exactly, and folds the scale away where it would show:
Mul after the embedding gather, since the table
is shared with the logits), and so are the out_proj and fc2 biases and
the fc2 weights.v_proj), so the probability
products and then the out_proj products are 2^10 larger. The value cache
(present.*.decoder.value) holds the scaled values and is fed back as is.LayerNormalization node (the graph declares
opset 17), because ONNX Runtime's pattern fusion reads a float16 epsilon
back as 1e-5 on its CPU provider.cross_attentions.* as
float32, and scales its probabilities by 2^10 before the float16 cast the
value product reads: the tail a float16 softmax flushes is kept down to
2^-24.The scale is a power of two and by itself changes no bits on a GPU that keeps
subnormals; the float32 softmax does, by one frame at two word boundaries of a
30 s window on the desktop. The final layer norm removes the scale: logits,
cross_attentions.* and cross_kv_fp16.onnx mean what they did. On the windows measured every scaled
tensor stays 13 times below the float16 maximum and any partial sum of a
scaled product 8 times below.
Against 4d1b2589, forced to the reference tokens of three 30 s windows, the
logits on both Adreno GPUs go from 1.7-2.0 to 0.3-0.5 from the CPU provider,
the desktop's 0.3-0.6. On a 10 s excerpt, token times read from the alignment
heads differ from the CPU provider's at 4 of 39 tokens by at most one 20 ms
frame, where 10 tokens moved by up to five; the desktop moves 4 tokens by up
to three. Decoding 30 s of a song with the musetric runtime, both phones now
return the desktop's words, and their word times differ from the desktop's by
one frame at two of 44 word boundaries. Speed and memory do not change. On the
desktop the word error rate over 13 openly licensed songs goes from 22.75 to
22.59 %.
What remains is float16 accumulation itself: every GPU sums these products in float16, so word times can differ by a few frames between GPUs and from the CPU provider.
The float16 step needs a host that runs cross_kv once per window, feeds the
prompt one token at a time and reads step_outputs; musetric packages/ai
is one; the stock pipeline of transformers.js is not. Each graph is created
with its weights, as in onnxruntime-web:
import * as ort from 'onnxruntime-web/webgpu';
const base =
'https://huggingface.co/musetric/whisper-large-v3-turbo-onnx/resolve/main';
const bytes = async (file: string): Promise<Uint8Array> =>
new Uint8Array(await (await fetch(`${base}/${file}`)).arrayBuffer());
const encoder = await ort.InferenceSession.create(
await bytes('encoder_model_q4.onnx'),
{
executionProviders: ['webgpu'],
externalData: [
{
path: 'encoder_model_q4.onnx_data',
data: await bytes('encoder_model_q4.onnx_data'),
},
],
},
);
Outside the browser, onnxruntime reads <graph>_data from beside the graph.
Intended: client/edge speech-to-text with word timings via WebGPU through
@huggingface/transformers — e.g. lyric alignment in musetric.
Limitations:
shader-f16.musetric pipeline adds language
forcing and loop guards on top.Documented only as far as it is verifiable.
openai/whisper-large-v3-turbo. Inference-only
re-export in the transformers.js layout; no fine-tuning.[layer, head] pairs come from the
generation_config.json of the base model and match the
whisper-large-v3-turbo entry of the transformers.js converter table.transformers==4.42.4 / optimum==1.21.3 /
torch==2.4.1 (transformers 4.43 adds a cache_position decoder input that
@huggingface/transformers 4.2.0 does not feed).scripts/onnx/whisper in musetric/lab;
the float16 decoder comes from fp16_decoder.py there, not from
quantize.py --modes fp16, which does not produce a loadable graph for a
model with If branches.onnx-community publishes a comparable
whisper-large-v3-turbo_timestamped; this repo exists so the export
pins, the file set and the checksums are the ones musetric produces itself.
MIT, following the model card the weights are downloaded from and the
openai/whisper repository, which states that "Whisper's code and model
weights are released under the MIT License". The model card declared
apache-2.0 only in its first days; its revision 41f01f3f of 2024-10-04
changed it to mit.