whisper-acft-ko

Whisper ACFT-KO — Korean audio-context fine-tuned short-window TFLite (LiteRT)

Korean audio-context fine-tuned (ACFT-KO) Whisper models, exported as fixed short-window TFLite (LiteRT) graphs with encode/decode-split signatures. Four model sizes (tiny / base / medium / large-v3-turbo), three window lengths each (5 s / 10 s / 30 s), all dynamic-range int8 quantized ("drq").

Trained and exported by the LiteRT-LM-Unity project for on-device Korean voice commands.

What ACFT-KO is

Audio-context fine-tuning per the method of futo-org/whisper-acft (method citation): Whisper is fine-tuned to tolerate a shortened encoder audio context, so short clips can be transcribed without padding to 30 s — cutting encoder cost by up to ~6-12x for voice-command use. Stock Whisper is trained for the 30 s window, so simply truncating the audio context puts it out of distribution (short-Korean CER 1.07-24.9 across sizes at a 5 s context); ACFT restores usable output at the short context.

These weights are not futo's checkpoints — they were trained from the stock openai checkpoints (whisper-tiny, whisper-base, whisper-medium, whisper-large-v3-turbo) by this project with a Korean-focused data mix.

Training

• Data: zeroth-korean (OpenSLR SLR40, CC-BY-4.0) 70 % + google/fleurs en_us (CC-BY-4.0) 30 %, with short-utterance augmentation (short clips oversampled/cropped to match the voice-command regime).
• Objective: MSE self-distillation on decoder hidden states — teacher = the same checkpoint at full 30 s context; student sees per-batch sampled audio contexts with a floor of n_audio_ctx = 250 (the fixed 5 s deployment window).
• lr 1e-6, early stopping on a Korean-short + FLEURS composite CER gate. All four models passed their training gates. The Korean-short bucket of that gate is TTS-synthesized — it ranks tiers correctly but is not an absolute quality measure, and passing it did not make every tier usable (see "Evaluation caveats & tier guidance").

Files

Converter-time dynamic-range quantization (int8 weights, fp32 activations — tf.lite.Optimize.DEFAULT).

Tokenizers are unchanged from the base models — use tokenizer.json from the corresponding openai checkpoint (tiny/base/medium: vocab 51865, 80 mel bins; large-v3-turbo: vocab 51866, 128 mel bins, <|transcribe|> = 50360, <|notimestamps|> = 50364).

REQUIRED integration note

These are fixed short-window graphs, not 30 s graphs. The encode signature input is [1, n_mels, frames] with frames = 500 / 1000 / 3000 for the 5 s / 10 s / 30 s files. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames. Also note the decode signature input order of these exports is (mask, audio, tokens) — opposite of the stock litert-community exports — so bind decode tensors by shape/name rather than position. Reference implementation: the take6 JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the signature on one code path.

The decoder is a fixed-length full re-run per step (no KV cache), matching the litert-community whisper-tiny/base graph interface.

⚠️ Evaluation caveats & tier guidance — read before using the numbers below

A 2026-07-26 methodology audit (inference-only re-evaluation of the checkpoints, 7 checkpoints × 7 eval buckets) re-scored these models on real human recordings instead of the synthesized gate set. Three things you need to know:

1. tiny is NOT recommended for Korean voice commands. Real-recorded-command CER 0.896; on device it got 1/4 commands exact and was rejected for deployment. The published gate number for tiny (0.457) reads as "workable small tier" and it is not one. Root cause: this tier was distilled from the stock openai/whisper-tiny checkpoint, so it is effectively an English model with fragile Korean — its real-recorded English CER is 0.031 (the best in the whole lineup) against 0.896 on Korean commands. Use tiny only for English/near-English short-window work, or pick base (fast path) / large-v3-turbo (accuracy path) for Korean commands.

2. The training gate is a TTS-synthesized bucket. It ranks models correctly; it does not give absolute quality. The gate is 40 edge-tts synthesized short-command clips (0.46–1.43 s). Against the real recordings its ranking is validated — Spearman ρ = 1.000, Pearson r = 0.992 across 7 checkpoints — so model-to-model comparisons made with it are trustworthy. Its absolute level is not calibrated: for a strong un-finetuned base it is ~2.8× harsher than real audio, and for this tiny tier it is optimistic (TTS 0.728 vs real 0.896 when both are measured at full context). Do not read a gate CER as an "is this shippable" threshold.

3. Punctuation sensitivity — the gate references average 3.6 characters. On such short references a single trailing period is +0.25 to +0.50 CER. In the audited comparisons, 33–79 % of the measured CER "regression" between checkpoints was trailing punctuation alone, and a typical voice-command matcher (including the reference implementation's) ignores punctuation and spacing entirely. Treat any absolute CER from a short-command bucket as an upper bound on product-relevant error, and score punctuation-stripped as well.

Context note for honesty about the tiny numbers: the published 0.457 was measured at the 5 s deployment context on 24 of the 40 gate clips, while the audit re-measured the same checkpoint at full 1500-frame context (TTS-40 0.728, real recorded commands 0.896, real spontaneous short Korean 0.937). So 0.457 → 0.896 is not a like-for-like delta — part of the gap is the context/subset difference. What is unambiguous is that every measurement of this tier on real Korean audio (either context, desktop or device) lands outside the range we would deploy for Korean voice commands. Only the tiny tier was re-measured on real recordings by the audit; base / medium / large-v3-turbo rest on their device results below and are unaffected by this correction.

Validation

Training gate — TTS-synthesized short-command eval (desktop, 5 s context) — ranking-valid, not an absolute quality number

Eval bucket = a 40-clip edge-tts synthesized short-command holdout (0.46–1.43 s, mean reference 3.6 characters); the gate scored 24 of the 40 clips (per-bucket SEM ≈ 0.05). CER vs punctuation-normalized references, greedy decode. "Stock" = the unmodified openai checkpoint evaluated at the same truncated 5 s audio context. Read the columns as an ordering of the tiers, not as expected real-world CER (see caveats above):

The same tiers against real audio

"Real-recorded command CER" = 2026-07-26 audit, checkpoint at full context on human recordings of the target commands. "Device" = these 5 s TFLite exports on a Snapdragon 865 (five device test cycles). Where the audit did not measure a tier, the device column is the real-audio evidence for it:

On device (Snapdragon 865, arm64 CPU, LiteRT, fresh process incl. model compile)

Korean voice-command clips + one 3.8 s Korean report sentence:

Desktop rebench of the exports (7-clip Korean/English matrix): medium-5s and turbo-5s/10s all-Korean-exact including the quiet take; per-window mean CER medium 0.039, turbo 0.031 (best of the queue).

Caveats

• The 5 s graph hard-truncates longer audio — use the 10 s/30 s variants for dictation/longer clips.
• tiny is NOT recommended for Korean voice commands — real-recorded-command CER 0.896, device 1/4 exact (rejected for deployment). Its TTS gate number (0.457) overstates its Korean ability; it was distilled from the stock English-leaning checkpoint (real-recorded English CER 0.031). Published for completeness / English-leaning small targets. Use base (fast path) or large-v3-turbo (accuracy path) for Korean.
• The training-gate CERs are TTS-synthesized and ranking-valid only — see "Evaluation caveats & tier guidance" above. Never quote them as absolute Korean quality, and note that on ~3.6-character references a single trailing period is +0.25–0.50 CER while most command matchers ignore punctuation.
• Sub-1.2 s very-quiet clips remain AGC-dependent (capture-side loudness normalization recommended) on the smaller models; large-v3-turbo-5s handles the quiet gate take.
• 10 s/30 s medium variants regress one command clip (음량 → 음향); 30 s turbo drops one word on one clip vs its 5 s/10 s windows. Pick the window that matches your audio length rather than defaulting to 30 s.

License / attribution

• tiny / base / medium files: Apache-2.0 (matching the upstream openai/whisper-tiny/base/medium releases).
• large-v3-turbo files: MIT (matching upstream openai/whisper-large-v3-turbo).
• Method: futo-org/whisper-acft (audio-context fine-tuning).
• Training data attribution:
  • Zeroth-Korean corpus (Lucas Jo & Wonkyum Lee, Atlas Guide) — OpenSLR SLR40, CC-BY-4.0.
  • FLEURS (Google) — google/fleurs, CC-BY-4.0.

Produced by LiteRT-LM-Unity.