whipa-base-cv

whipa-base-cv

This Whisper-for-IPA (WhIPA) model is a fine-tuned version of openai/whisper-base on a subset of the CommonVoice11 dataset (1k samples each from Greek, Finnish, Hungarian, Japanese, Maltese, Polish, Tamil) with G2P-based IPA transcriptions. It (Ckpt4) achieves the following results on the evaluation set:

• Train Loss: 0.7632
• Validation Loss: 1.0754
• Lvnshtn: 19.1200
• Cer: 0.5521
• Cer Norm: 0.4570
• Ped: 17.9029
• Per: 0.5793
• Pfer: 18.4631
• Levenshtein phone distance (multipa): 6.4574
• Weighted feature edit distance (Panphon): 63.6318
• Weighted feature edit rate (Panphon): 1.6310

Model description

For deployment and description, please refer to https://github.com/jshrdt/whipa.

from transformers import WhisperForConditionalGeneration, WhisperTokenizer, WhisperProcessor

whipa_model = WhisperForConditionalGeneration.from_pretrained("jshrdt/whipa-base-cv/checkpoint-440") # recommended checkpoint

whipa_model.generation_config.language = "<|ip|>"
whipa_model.generation_config.task = "transcribe"

whipa_tokenizer = WhisperTokenizer.from_pretrained("jshrdt/whipa-base-cv", task="transcribe")
whipa_processor = WhisperProcessor.from_pretrained("jshrdt/whipa-base-cv", task="transcribe")

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 64
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 110
• training_steps: 1100
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.48.3
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0