asr-laughter-whisper

Non-lexical sounds such as laughter are considered important discourse markers in conversation as these sounds complement the lexical information in shaping interpersonal relations, managing the conversation, and in expressing attitudes and affect. However, general purpose open-source automatic speech recognition (ASR) systems are typically not focusing on these sounds. Laughter transcription is an understudied task in ASR: laughter is often not modelled as a token and is discarded in the evaluation of ASR. In our study, we investigate how current open-source ASR models are handling laughter and speech-laugh (speech interspersed with laughter). Using Switchboard and BuckEye as conversational speech corpora, we fine-tuned and evaluated Whisper and Wav2Vec2. Our results show that laughter can be integrated in ASR transcriptions without substantially degrading word error rate.

Ho, P.H., Bălan, D.A., Heylen, D.K.J., Truong, K.P. (2025) Enhancing Transcripts of Open-Source Automatic Speech Recognition Models Through Fine-Tuning with Laughter and Speech-Laugh. Proc. Interspeech 2025, 4513-4517, doi: 10.21437/Interspeech.2025-2193

Usage

from transformers import AutoProcessor, WhisperForConditionalGeneration
import torch
import librosa

# Path to the local model folder
model_name = 'speechlaugh-whisper/'

model = WhisperForConditionalGeneration.from_pretrained(
    model_name, dtype=torch.float32, device_map="cpu"
)

audio, sr = librosa.load("demo_input.wav", sr=16000)

inputs = processor(
    audio,
    sampling_rate=16000,
    return_tensors="pt"
)

pred_ids = model.generate(inputs.input_features)

hyp = processor.tokenizer.decode(pred_ids[0], skip_special_tokens=True)

print(f"HYP: {hyp}")

Requirements

Tested with:

accelerate==1.14.0
python==3.12.13
librosa==0.11.0
torch==2.13.0
transformers==5.13.1