crisper-whisper2-base-finetuned-ko

CrisperWhisper2 Korean Verbatim (base, 74M)

Korean verbatim ASR (74M), the base of a three-size family. Transcribes spontaneous Korean as spoken, keeping the fillers and repeats (어, 음, 그, 막 ...) that standard ASR drops. Built by transferring CrisperWhisper2.0's verbatim strength (an English-only model) into Korean.

CER 0.167 at 74M, beating whisper-small (244M, 0.171) at one third the size.

Non-commercial / research use only. Derived from a non-commercial teacher (CrisperWhisper2.0) and restricted data (KsponSpeech). See Licensing below.

Model family

Method

This base model is soft-KD distilled from our Korean small model (there is no CrisperWhisper checkpoint at this size to initialize from, so the advantage is transferred by distillation).

Results: vs Whisper and CrisperWhisper

Held-out KsponSpeech valid, 300 spontaneous Korean clips. CER lower is better; disfluency = fillers kept; tag-leak = leaked markup tokens (0 is clean).

CrisperWhisper2.0_small is run with its 31 markup tokens suppressed, so it emits clean text instead of looping on [verbatim_N]. This barely changes its CER (0.68 vs 0.71): it is English-centric and does Korean poorly either way.

What it shows:

• Our base (74M) beats whisper-small (244M) on Korean (0.167 vs 0.171) at one third the size.
• CrisperWhisper-init (0.103) vs whisper-init (0.265), same data: the CrisperWhisper prior transfers to Korean and cuts CER 2.6x. Both reached the same training loss, so the gain is generalization.
• base-KD (74M) beats the whisper-init 244M ablation (0.265): the advantage propagates down via distillation.
• Our models keep 86 to 92% of disfluencies. The original CrisperWhisper, even markup-suppressed, sits at CER 0.682 on Korean: it drops clauses, hallucinates English, or repeats characters.

Transcription examples

Complete outputs on held-out clips (nothing trimmed). REF is the human verbatim transcript; CrisperWhisper is the fair markup-suppressed run.

Example 1: a long turn opening with the filler 「음」

REF             음 그리고 인디 게임 아니 인디 게임이래 인디 음악도 좋아한다고 하더라고 그래서 들어봐야 되는데 그거 아직까지 못 들어봤어 오늘 집에 가면서 꼭 들어봐야지. 응.
small (ours)    음 그리고 인디 게임 아니 인디 게임이래. 인디 음악도 좋아한다고 하더라고. 그래서 들어봐야 되는데 그거 아직까지 못 들어봤어. 오늘 집에 가면서 꼭 들어봐야지. 응.
base  (ours)    응. 그리고 인디 게임 아니 인디 게임이라 인디 음악도 좋아한다고 하더라고. 그래서 들어봐야 되는데 그 아직까지 못 들어봤어. 오늘 집에 가면서 꼭 들어봐야지. 응.
tiny  (ours)    음 그리고 인디 게임 아니 인디 게임이라 인디 음악도 좋아한다고 하더라고 그래서 들어봐야 되는데 그거 아직까지 못 들어봤어. 오늘 집에 가면서 꼭 들어봐야지. 응.
whisper-small   그리고 인디게임.. 아니 인디게임이래 인디 음악도 좋아한다고 하더라고 그래서 들어봐야 되는데 아직까지 못 들어봤어 오늘 집에 가면서 꼭 들어봐야지
CrisperWhisper  그리고 인디 음악도 좋아한다 하더라고? 그래서 들어봐야 하는데 아직까지 못 들어봤어. 오늘 집에 가면서 꼭 들어봐야지.

What differs: our models reproduce the full turn, opening filler 「음」 and closing 「응」 included. whisper-small drops the 「음」, 「그거」, and the final 「응」. CrisperWhisper drops the whole 「인디 게임 아니 인디 게임이래」 clause.

Example 2: filler 「어」 plus a self-repeat 「나 닌텐도 ... 나 닌텐도」

REF             어 닌텐도 맨 처음에 나 닌텐도 맨 처음에 나올 때 그런 건데 아 근데 진짜 좀 여
small (ours)    어. 닌텐도 맨 처음에 나 닌텐도 맨 처음에 나올 때 그런 건데 아 근데 진짜 좀 여
base  (ours)    어 닌텐도 맨 처음에 나 닌텐도 맨 처음에 나올 때 그런 건데 아 근데 진짜 좀 여행
tiny  (ours)    어. 닌텐도 맨 처음에는 닌텐도 맨 처음에는 할 때 그러는데 아 근데 진짜 좀 여
whisper-small   닌텐도 맨 처음에 나올 때 그런 건데 아 근데 진짜 좀 여..
CrisperWhisper  'Nintendo Manchera' 맨날 'Nintendo Manchera' 나올 때 그런 건데, 아, 근데 진짜 좀,

What differs: our small/base keep the filler 「어」 and the self-repeat 「나 닌텐도 ... 나 닌텐도」. whisper-small drops the 「어」 and collapses the repeat. CrisperWhisper hallucinates the Korean word 「닌텐도」 into English "Nintendo Manchera".

Example 3: a short turn that is mostly the filler 「아」

REF             아 아 아 톤이 높아요? 아
small (ours)    아 아 아 톤이 높아요? 아
base  (ours)    아 아 아 토익 높아요? 어.
tiny  (ours)    아 아 토익도 벌어?
whisper-small   아, 아, 아, 아, 톤이 높아요?
CrisperWhisper  -ㅎㅎ. -ㅎㅎ.

What differs: our small reproduces it exactly, all four 「아」 and 「톤이 높아요」. whisper-small adds a fourth 「아」 and drops the trailing one. CrisperWhisper degenerates into "-ㅎㅎ.". (On this very short clip our base and tiny mishear 「톤이」 as 「토익」, an honest small-model slip.)

Rare-word slips are real (「톤이」 to 「토익」) and we do not hide them (our CER is 0.10 to 0.22). The point is which model preserves how the sentence was actually spoken, fillers and repeats included.

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import librosa, torch

repo = "Sky-Kim/crisper-whisper2-base-finetuned-ko"
model = WhisperForConditionalGeneration.from_pretrained(repo).eval()
proc  = WhisperProcessor.from_pretrained(repo)

audio, _ = librosa.load("your_korean.wav", sr=16000, mono=True)
feat = proc.feature_extractor(audio, sampling_rate=16000, return_tensors="pt").input_features
ids  = model.generate(feat, language="ko", task="transcribe", max_new_tokens=200)
print(proc.tokenizer.decode(ids[0], skip_special_tokens=True))

Architecture: Whisper 512 dim, 6+6 layers with the CrisperWhisper 51,896-token vocabulary. Emits plain verbatim text (no markup tags). On-device target: Unity Sentis 2.6 (ONNX / Float16 export).

Training data

Labels are KsponSpeech ground truth (markup parsed to verbatim text, fillers kept). Korean only, encoder frozen. small: 15k steps. base and tiny: soft-KD from small, 12k steps. Eval is the held-out KsponSpeech valid split.

Licensing and redistribution

Effective license is the most restrictive of the components, so this is non-commercial / research use only.

• CrisperWhisper2.0 (nyralabs), used as the small init, the KD teacher for base and tiny, and the tokenizer, is under the Nyra Health Non-Commercial Research License (commercial use requires a commercial license). These weights inherit that restriction.
• KsponSpeech (AI Hub / NIA), the Korean training labels, is not an open license. Redistribution and commercial use are restricted by the AI Hub Terms of Use.
• Zeroth-Korean is CC BY 4.0 (attribution). Whisper skeletons are MIT.

Commercial use requires a commercial license from Nyra Labs plus clearing the AI Hub KsponSpeech terms (or retraining with permissive components). License chain recorded to our knowledge (2026-08); not legal advice. Verify current terms before distributing. Full attribution in NOTICE.md.

Limitations

Korean only (no English). Verbatim output keeps disfluencies, which raises CER against cleaned references but is the point for verbatim use (meeting notes, speech analysis, subtitles as said). tiny is less accurate than base and small. Numbers are in-domain (KsponSpeech).