whisper-acft

Whisper ACFT — short-window TFLite (LiteRT), tiny/base/small × multilingual/.en

TFLite (LiteRT) exports of the FUTO audio-context fine-tuned (ACFT) Whisper checkpoints: fixed 5 s / 10 s / 30 s window graphs split into encode and decode signatures, matching the graph interface of litert-community/whisper-tiny and litert-community/whisper-base. Six models in one repo: tiny, base, small (multilingual) and tiny.en, base.en, small.en (English-only), each in three window lengths.

No retraining — the weights are FUTO's Apache-2.0 ACFT checkpoints (futo-org/whisper-acft method and training; HF checkpoints futo-org/acft-whisper-{tiny,base,small}{,.en}). This repo only converts them: transformers TFWhisperForConditionalGeneration → two-signature graph → converter-time dynamic-range quantization (int8 weights, fp32 activations, tf.lite.Optimize.DEFAULT). Conversion by the LiteRT-LM-Unity project.

Why short windows

Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window — exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders are trained on the full 30 s window, so shrinking it puts them out of distribution (repetition loops: stock base at a 5 s window measured CER 10.4 on a sentence clip in our evaluation); the ACFT checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this.

Repo layout

One folder per model; filenames encode model and window. All files are dynamic-range quantized (int8 weights, fp32 activations).

Window ↔ encode input shape: 5 s = [1, 80, 500], 10 s = [1, 80, 1000], 30 s = [1, 80, 3000].

Integration notes

• Fixed short-window graphs: the encode signature input is [1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames.
• The decode signature input order is (mask, audio, tokens) — opposite of the stock litert-community exports — bind decode tensors by shape/name rather than position. The decoder is a fixed-length full re-run per step (no KV cache), matching the tiny/base graph interface.
• Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the encode signature on one code path.

Token ids — multilingual vs .en

• Multilingual (tiny/, base/, small/): 80 mel bins / vocab 51865 — classic Whisper token layout, same as litert-community tiny/base. Decode prompt e.g. [<|startoftranscript|>=50258, <|lang|>, <|transcribe|>=50359, <|notimestamps|>=50363]. Tokenizer: tokenizer.json from the matching openai/whisper-* repo.
• English-only (tiny.en/, base.en/, small.en/): 80 mel bins / vocab 51864 — English-only layout: <|startoftranscript|> = 50257, <|endoftext|> = 50256, <|notimestamps|> = 50362; there are no language/task tokens — the decode prompt is [50257, 50362]. Tokenizer: tokenizer.json from the matching openai/whisper-*.en repo.

Validation

Desktop probe of these files (LiteRT interpreter, XNNPACK CPU, greedy decode; CER vs punctuation-normalized references, whitespace removed). Multilingual clips: Korean sentence + English sentence + short Korean voice commands; .en clips: English sentence pair.

tiny

The short windows track the checkpoint's own 30 s behavior (same clips right/wrong at every window; residual errors are tiny's model-capacity errors, present at 30 s too) at ~14x encoder speedup for the 5 s window.

tiny.en

Transcript-identical to the 30 s window on both clips (including the same "and soul" mishearing — a tiny-capacity trait, present at 30 s too) at ~13x encoder speedup for the 5 s window.

base

~11x encoder speedup for the 5 s window; base is the recommended minimum for non-English voice commands.

base.en

small

12/12 exact across all three windows — the 5 s window is character-identical to the 30 s window at ~10x encoder speedup.

small.en

All exact. In the same evaluation pipeline on whisper-base, the ACFT 5 s graph reproduced the stock 30 s transcripts clip-for-clip at ~8x encoder speedup (f32) / ~20x (drq); stock checkpoints at a 5 s window (no ACFT) fall into a repetition loop on sentence-length clips (CER 10.4). Short-window graphs are device-validated on Android arm64 (Snapdragon 865, 0.5–0.8 s end-to-end per command clip for tiny/base 5 s) in the producing project.

Caveats

• The 5 s/10 s graphs hard-truncate longer audio — pick the window that matches your audio length; use the 30 s graph for dictation.
• Decoder is a full-sequence re-run per step (no KV cache) to match the litert-community tiny/base interface — a KV-cached runtime will be substantially faster per token.
• tiny retains its stock model-capacity limits on short noisy/quiet clips; base is the recommended minimum for non-English voice commands.
• Language forcing recommended for short clips on the multilingual models (e.g. <|ko|> / <|en|>).

Credits and license

• ACFT method and checkpoints: FUTO (futo-org/acft-whisper-*, Apache-2.0, derived from the MIT openai/whisper-* checkpoints). This repo contains TFLite conversions of those checkpoints, no retraining.
• Conversion and validation: LiteRT-LM-Unity.
• License: Apache-2.0, matching the source ACFT checkpoints.

Related: Korean-fine-tuned ACFT short-window models (tiny/base/medium/large-v3-turbo) live at leuconoe/whisper-acft-ko.