whisperkit-coreml

VocaMac WhisperKit models

Community Whisper fine-tunes converted to WhisperKit CoreML, so they run on-device on Apple Silicon: on the Neural Engine, offline, with no audio leaving the Mac. They power the extra languages in VocaMac, an open-source macOS dictation app, and any other WhisperKit app can load them too.

The layout matches argmaxinc/whisperkit-coreml: one folder per model, with the tokenizer and configs inside it.

Models

Which one? Use Voca Hinglish if you want Hindi written the way people text it (kal meeting hai, please calendar update kar do). It is smaller, faster, and the one VocaMac ships. Use the medium model if you want Devanagari output.

Quick start

In VocaMac

Open Settings → Speech Model and pick Voca Hinglish. VocaMac downloads it, pins the decoder language, and filters out decoder loops for you.

In your own app

Add the argmax-oss-swift package (product WhisperKit), then:

import WhisperKit

let whisper = try await WhisperKit(WhisperKitConfig(
    model: "vocahq_voca-hinglish_820MB",
    modelRepo: "VocaHQ/whisperkit-coreml"
))

let options = DecodingOptions(
    language: "en",          // required: this is what makes it write Hinglish
    temperature: 0,
    usePrefillPrompt: true,
    detectLanguage: false
)
let results = try await whisper.transcribe(audioPath: "voice-note.wav", decodeOptions: options)
print(results.map(\.text).joined(separator: " "))

WhisperKit downloads the folder on first use. To fetch it ahead of time, for bundling or offline machines:

hf download VocaHQ/whisperkit-coreml --include "vocahq_voca-hinglish_820MB/*" --local-dir ./models

Then point WhisperKit at it with WhisperKitConfig(modelFolder: "./models/vocahq_voca-hinglish_820MB").

Voca Hinglish

Speak Hindi, get Hinglish: Hindi speech written in Roman script, the way people actually type it. English stays English, and sentences that switch between the two come out as spoken.

Voca Hinglish is the VocaMac build of Oriserve/Whisper-Hindi2Hinglish-Apex (Apache-2.0), a fine-tune of Whisper large-v3-turbo on about 700 hours of noisy, Indian-accented Hindi. All credit for the model goes to Oriserve; VocaHQ only converted and compressed it. Oriserve reports 29.8% WER on FLEURS hi_in and 36.0% on Common Voice 20 Hindi for the original model; see their card for details.

Language setting matters

The model was trained to write Hinglish when decoding as English. Force language: "en" and turn language detection off. Asked for Hindi, or left to detect, it falls back to Devanagari or translates to English.

Fidelity to the original

Compression was checked against the unquantized fp16 CoreML conversion on 60 clips of FLEURS hi_in test:

A 6-bit build was also tried and rejected: 3.4% WER against fp16.

First load and memory

CoreML compiles the model for the Neural Engine the first time it loads. On an M1 Pro that took about 5 minutes and peaked at 3.1 GB of memory, most of it in macOS's ANECompilerService rather than the app, because the compiler expands the palettized weights. Later loads come from the compile cache and take seconds.

Known limitations

• Romanization is informal. Hindi words are spelled the way people type them, which varies (nahi / nahin, thik / theek). It is not a standard transliteration scheme.
• Occasional Devanagari. A word or phrase now and then comes out in Devanagari instead of Roman script.
• Rare decoder loops. Like other Whisper models, it can repeat a phrase or a single token at the end of a long utterance, sometimes in an unrelated script (one dictation ended in в к followed by ~170 т). VocaMac detects loops and removes letters outside Latin and Devanagari; if you use the model directly, filter the output the same way.
• Hindi only. Other Indian languages are not supported.

Conversion details

• Converted with whisperkittools using uniform 8-bit palettization. Argmax's mixed-bit recipe search was tried first and failed on this model.
• generation_config.json gains the alignment_heads of openai/whisper-large-v3-turbo, which the source repo omits and the converter requires.

whisper-medium-hindi-hinglish

Converted from shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned (Apache-2.0), a fine-tune of openai/whisper-medium for Hindi and Hinglish, trained on synthetic conversational speech. All credit for the weights goes to the original author; this repository only republishes them in the format WhisperKit loads. It is not in the VocaMac model catalog.

Choosing a language

The fine-tune expects Hindi or Hinglish input and is sensitive to the language you force on it. On a Hinglish sentence ("Kal ka meeting postpone ho gaya hai, so please update the calendar"):

Use Hindi for Devanagari output and auto-detect for English output. Forcing English can return nothing at all. Plain English speech transcribes correctly under all three.

Conversion details

• Converted with whisperkittools at fp16.
• The source checkpoint's tokenizer_config.json stores extra_special_tokens as a list, which current transformers rejects. The key is dropped here, since those tokens are already in tokenizer.json.

License and credits

Each model keeps the license of its source checkpoint. Both are Apache-2.0.

• Voca Hinglish: Oriserve, from Whisper-Hindi2Hinglish-Apex (GitHub)
• whisper-medium-hindi-hinglish: shujaAK
• Whisper: OpenAI
• WhisperKit and whisperkittools: Argmax

Found a problem with a conversion, or want another fine-tune converted? Open an issue on VocaHQ/vocamac.