whisper-th-medium-timestamp

Whisper Medium (Thai) Timestamp

This model is a fine-tuned version of biodatlab/whisper-th-medium-combined on a custom-created longform dataset derived from the CMKL/Porjai-Thai-voice-dataset-central. It achieves the following results on the common-voice-13 test set:

• WER: 15.57 (with Deepcut Tokenizer)

Model description

This model is designed to perform automatic speech recognition (ASR) for the Thai language, with the added capability of generating timestamps for the transcribed text. It's based on the Whisper medium architecture and has been fine-tuned on a specially crafted dataset to enable timestamp generation.

Use the model with Hugging Face's transformers as follows:

from transformers import pipeline
import torch

MODEL_NAME = "biodatlab/whisper-th-medium-timestamp"  # specify the model name
lang = "th"  # Thai language

device = 0 if torch.cuda.is_available() else "cpu"

pipe = pipeline(
    task="automatic-speech-recognition",
    model=MODEL_NAME,
    chunk_length_s=30,
    device=device,
    return_timestamps=True,
)
pipe.model.config.forced_decoder_ids = pipe.tokenizer.get_decoder_prompt_ids(
  language=lang,
  task="transcribe"
)
result = pipe("audio.mp3", return_timestamps=True)
text = result["text"]
timestamps = result["chunks"]

Intended uses & limitations

This model is intended for Thai automatic speech recognition tasks, particularly where timestamp information is required. It can be used for transcribing Thai audio content, creating subtitles, or any application that needs to align text with specific time points in audio. The model's performance on speech recognition may be lower compared to non-timestamped versions due to the additional complexity of the task and the pseudo-timestamp generation method used in training.

Training and evaluation data

The model was trained on a custom-created longform dataset derived from the CMKL/Porjai-Thai-voice-dataset-central. The dataset creation process involved the following steps:

• Combining multiple short audio clips from the original dataset into longer audio segments (up to 30 seconds).
• Adding environmental noises and silences between clips to simulate more realistic speech scenarios.
• Generating pseudo-timestamps for the combined audio using a Voice Activity Detection (VAD) model (Silero VAD).

This approach allowed us to create a dataset with longer, more diverse audio samples and approximate timestamp information, which is crucial for training a model capable of generating timestamps.

Training procedure

The model was fine-tuned using a custom training script that incorporates the following:

• Mixed precision training (FP16)
• Gradient accumulation
• SpecAugment for data augmentation during training

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• gradient_accumulation_steps: 1
• num_train_iters: ~50000
• warmup_steps: 500
• fp16: True
• optimizer: AdamW
• lr_scheduler_type: linear

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1
• Datasets 3.0.0
• Tokenizers 0.20.0

Performance and Limitations

The WER (Word Error Rate) of 15.57 on the Common Voice 13 test set indicates good performance for Thai ASR. However, it's important to note that the timestamp generation model has a lower accuracy compared to the non-timestamped version of the model. This is due to several factors:

• The use of pseudo-timestamps in training data, which are approximations based on VAD rather than precise human annotations.
• The additional complexity of the timestamp prediction task, which requires the model to learn both transcription and temporal alignment.
• Potential discrepancies between the VAD-generated timestamps and actual word boundaries in continuous speech.

Users should be aware that while the timestamps provide a general indication of when words or phrases occur in the audio, they may not be as precise as manually annotated timestamps. The model's performance may also vary depending on the acoustic conditions, speaker variability, and the presence of background noise in the input audio.

Citation

If you use this model in your research or applications, please cite it as follows:

@misc{biodatlab_whisper_th_medium_timestamp,
    author       = {Atirut Boribalburephan, Zaw Htet Aung, Knot Pipatsrisawat, Titipat Achakulvisut},
    title        = {Whisper Medium Thai Timestamp: A fine-tuned Whisper model for Thai automatic speech recognition with timestamp generation},
    year         = 2024,
    publisher    = {Hugging Face},
    howpublished = {\url{https://huggingface.co/biodatlab/whisper-th-medium-timestamp}}
}