nb-whisper-dialect-id-4dialect-low-pass

nb-whisper-dialect-id-4dialect-low-pass

A NbAiLab/nb-whisper-medium encoder fine-tuned for 4-class Norwegian dialect identification on low-pass filtered speech.

Given an audio clip of low-pass filtered Norwegian speech, the model predicts which of four broad dialect regions the speaker belongs to:

This model is one of three "prosody-condition" models (unmodified / low-pass / monotonize) trained for:

Phoebe Parsons, Heming Strømholt Bremnes, Knut Kvale, Torbjørn Svendsen, and Giampiero Salvi. (2025). Effects of Prosodic Information on Dialect Classification Using Whisper Features. In Proceedings of Interspeech 2025, pages 2785–2789. doi: 10.21437/Interspeech.2025-200

The other two conditions:

• scribe-project/nb-whisper-dialect-id-4dialect — unmodified audio
• scribe-project/nb-whisper-dialect-id-4dialect-monotonize — F0-monotonized audio

See the did_prosody_whisper repo for the training/evaluation code and the audio manipulation scripts used for the paper.

Model description

The model is NbAiLab/nb-whisper-medium's encoder with a classification head on top (WhisperForAudioClassification), fully fine-tuned (no layers frozen) for sequence classification over 4 dialect labels. Unlike the unmodified-audio model, this model was trained and evaluated on speech that was low-pass filtered relative to each utterance's average F0. Low-pass filtering degrades segmental/phonetic intelligibility while retaining the F0 (pitch) contour, isolating prosodic cues from lexical/phonetic content.

Intended uses & limitations

Intended for research on Norwegian dialect identification, specifically on studying how much dialect information is carried by prosody alone. Audio fed to this model should be low-pass filtered the same way as during training (see the did_prosody_whisper repo) — running unmodified audio through this model does not reproduce the paper's "unmodified" condition results; use scribe-project/nb-whisper-dialect-id-4dialect for that. It is not intended for consequential decisions about individuals (e.g. hiring, legal, or identity-verification contexts).

Known limitations:

• Trained only on adult parliamentary speech; accuracy on other domains (conversational speech, children, non-native speakers) is not guaranteed.
• This repo ships the checkpoint from the final training epoch (epoch 3), for consistency with the unmodified and monotonize models in this series. In this case the final-epoch checkpoint was also the best-eval-accuracy checkpoint.
• The 4 dialect categories are coarse regional groupings, not a fine-grained dialect taxonomy, and boundaries between adjacent regions are inherently fuzzy.

How to use

import torch
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
import librosa

model_id = "scribe-project/nb-whisper-dialect-id-4dialect-low-pass"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModelForAudioClassification.from_pretrained(model_id)

# audio should be low-pass filtered the same way as during training — see the
# did_prosody_whisper repo for the filtering procedure
audio, sr = librosa.load("path/to/low_pass_audio.wav", sr=feature_extractor.sampling_rate, mono=True)

inputs = feature_extractor(
    audio,
    sampling_rate=feature_extractor.sampling_rate,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = torch.argmax(logits, dim=-1).item()
print(model.config.id2label[predicted_id])

Audio should be mono, resampled to 16kHz. Clips longer than 30 seconds were randomly subsampled to 30 seconds during training.

Training and evaluation data

Trained on the SSC (Storting/Parliament Speech Corpus), using a speaker-disjoint train/validation split (no speaker overlap between train and eval). Each example is a single-speaker audio segment labeled with one of the 4 dialect categories above. Prior to training, audio was low-pass filtered relative to the per-utterance average F0 (estimated with REAPER); see the did_prosody_whisper repo for the exact Praat scripts used.

Training procedure

Training hyperparameters

• learning_rate: 3e-05
• train_batch_size: 16 (per device)
• eval_batch_size: 32 (per device)
• seed: 0
• distributed_type: multi-GPU
• num_devices: 6
• gradient_accumulation_steps: 2
• total_train_batch_size: 192
• optimizer: Adam (betas=(0.9, 0.999), epsilon=1e-08)
• lr_scheduler_type: linear, warmup_ratio 0.1
• num_epochs: 3.0
• mixed_precision_training: Native AMP
• feature encoder: not frozen (full fine-tune)

Training results

The weights published in this repo are from epoch 3 (checkpoint-1356), the final training checkpoint — which in this condition also had the highest eval accuracy.

Framework versions

• Transformers 4.40.0
• PyTorch 2.2.2+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1