whisper-small-hebrew

Whisper Small - Hebrew Fine-tuned

Fine-tuned openai/whisper-small for Hebrew automatic speech recognition (ASR).

Model Details

• Base model: openai/whisper-small (244M params)
• Language: Hebrew
• Training data: ~50 hours of Hebrew speech (crowd-transcribe-v5 dataset)
• Training: 3 epochs, decoder-only fine-tuning (encoder frozen), bf16
• Final train loss: 0.29
• Final eval loss: 0.418

Training Configuration

Usage

With Transformers

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

processor = WhisperProcessor.from_pretrained("dm15/whisper-small-hebrew")
model = WhisperForConditionalGeneration.from_pretrained("dm15/whisper-small-hebrew")

# Transcribe
input_features = processor(audio_array, sampling_rate=16000, return_tensors="pt").input_features
predicted_ids = model.generate(input_features, language="he", task="transcribe")
text = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

With faster-whisper (recommended for production)

Convert to CTranslate2 first:

ct2-whisper-converter --model dm15/whisper-small-hebrew --output_dir whisper-hebrew-ct2 --quantization float16

Then use faster-whisper for 4-5x faster inference:

from faster_whisper import WhisperModel

model = WhisperModel("whisper-hebrew-ct2", device="cuda", compute_type="float16")
segments, info = model.transcribe("audio.wav", language="he", beam_size=1)
text = " ".join(s.text for s in segments)

Performance

• GPU inference (faster-whisper, RTX 3090): ~100-200ms per 3-5s utterance (RTF 0.03-0.10)
• CPU inference (faster-whisper, INT8): ~300-500ms per 3-5s utterance
• Trained on 16kHz audio. For telephony (8kHz), a separate fine-tuned version is recommended.