whisper-base.en-timestamped-onnx

whisper-base.en-timestamped-onnx

An ONNX conversion of openai/whisper-base.en for Transformers.js, exported so that the decoder emits cross-attention outputs, which is what word-level timestamps (return_timestamps: 'word') are computed from in the browser.

This repository contains no new model weights. The weights are OpenAI's, converted to the ONNX graph format and repackaged. All modelling credit belongs to OpenAI; see Attribution below.

Why this exists

A stock Optimum export of Whisper does not declare the decoder's attention tensors as graph outputs. Transformers.js reads word timings out of cross_attentions.* in WhisperForConditionalGeneration._extract_token_timestamps, so with a stock export it fails with:

Model outputs must contain cross attentions

This export declares cross_attentions.{0..5} (plus decoder_attentions.* and encoder_attentions.*) and traces the model with output_attentions=True so those outputs are actually written, not merely declared.

Usage

import { pipeline } from '@huggingface/transformers';

const transcriber = await pipeline(
  'automatic-speech-recognition',
  'mgndxb/whisper-base.en-timestamped-onnx',
  {
    revision: '<pin an immutable commit sha>',
    dtype: { encoder_model: 'fp32', decoder_model_merged: 'fp32' },
  },
);

const out = await transcriber(audio, {
  return_timestamps: 'word',
  chunk_length_s: 30,
  stride_length_s: 5,
});

generation_config.json carries the alignment_heads this checkpoint needs — [[3, 3], [4, 7], [5, 1], [5, 5], [5, 7]] — taken unmodified from the upstream repository. Word timestamps are unavailable without them.

Contents

fp32 only. Nothing here is quantized.

MANIFEST.json in this repository records the same hashes alongside the exact tool versions used, and is generated by the conversion script rather than written by hand.

Source and exact conversion procedure

Upstream model: openai/whisper-base.en Upstream revision: 911407f4214e0e1d82085af863093ec0b66f9cd6 Upstream licence: Apache-2.0

Every file here derives from that revision. Nothing was copied from any third-party ONNX conversion.

Toolchain

Procedure

optimum.exporters.onnx.main_export with task automatic-speech-recognition-with-past, model_kwargs={"output_attentions": True}, and a WhisperOnnxConfig subclass that adds the attention graph outputs. The three submodel configs use Optimum's own behaviour split — encoder; decoder (use_past=True, use_past_in_inputs=False); decoder-with-past (use_past=True, use_past_in_inputs=True) — after which Optimum merges the two decoders into decoder_model_merged.onnx. Unmerged intermediates are discarded.

The custom-config approach follows scripts/extra/whisper.py from Transformers.js v2.17.2 (Apache-2.0), re-implemented against a current Optimum.

The conversion script, the structural verifier, and a pinned requirements file are published alongside the application that uses this model.

Verification

Checked before publication:

• All six cross_attentions.* outputs are present in decoder_model_merged.onnx, populated by an ONNX Runtime run with shape (batch, 8 heads, decoder_len, 1500), and each row sums to 1 as a softmax should — a declared-but-never-written output would pass a name check and fail in production.
• Every alignment_heads entry indexes a layer and head that exist.
• End-to-end in a real browser via @huggingface/transformers, on both the WebGPU and the WASM backend, against 15–30s of real English speech: the pipeline loads, return_timestamps: 'word' succeeds with no cross-attention error, and word timestamps are monotonic and inside the clip.

Known limitations

• English only. whisper-base.en is an English-only checkpoint. It does not transcribe or translate other languages, and will emit confident nonsense if given them.
• Hallucination on non-speech. Like all Whisper checkpoints, it produces fluent, repeated phrases over silence, music and room tone. Gate it on a voice-activity check; do not present its output as a transcript unconditionally.
• base is the small end of the family. Expect noticeably worse accuracy than small/medium on accented speech, overlapping speakers, and noise.
• fp32 only, ~400MB. A first load is a large download. Quantized variants are deliberately absent: q8/q4 decoder weights of cross-attention Whisper exports have failed to load on the ONNX Runtime WASM backend (Missing required scale ... DequantizeLinear). Add them only behind their own parity run.
• Word timings are attention-derived, not forced-alignment. They are good enough to drive captions and drift on fast or slurred speech.
• Not evaluated for medical, legal, or safety-critical transcription.

Licence and attribution

Licensed under Apache-2.0, the licence of the upstream model this is converted from.

The weights are the work of OpenAI, published as openai/whisper-base.en under Apache-2.0. This repository claims no ownership of the model or its weights. It is a format conversion and packaging of that model: the same parameters, in ONNX, with attention outputs exposed. CreatorMission's contribution is the conversion configuration and packaging, offered under the same licence.

Whisper is described in Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022).

@misc{radford2022whisper,
  title  = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author = {Alec Radford and Jong Wook Kim and Tao Xu and Greg Brockman
            and Christine McLeavey and Ilya Sutskever},
  year   = {2022},
  eprint = {2212.04356},
  archivePrefix = {arXiv}
}

See NOTICE for the attribution notice required by Apache-2.0 §4(d).