nb-whisper-dialect-id-4dialect-monotonize

nb-whisper-dialect-id-4dialect-monotonize

A NbAiLab/nb-whisper-medium encoder fine-tuned for 4-class Norwegian dialect identification on F0-monotonized speech.

Given an audio clip of F0-monotonized Norwegian speech, the model predicts which of four broad dialect regions the speaker belongs to:

This model is one of three "prosody-condition" models (unmodified / low-pass / monotonize) trained for:

Phoebe Parsons, Heming Strømholt Bremnes, Knut Kvale, Torbjørn Svendsen, and Giampiero Salvi. (2025). Effects of Prosodic Information on Dialect Classification Using Whisper Features. In Proceedings of Interspeech 2025, pages 2785–2789. doi: 10.21437/Interspeech.2025-200

The other two conditions:

• scribe-project/nb-whisper-dialect-id-4dialect — unmodified audio
• scribe-project/nb-whisper-dialect-id-4dialect-low-pass — low-pass filtered audio

See the did_prosody_whisper repo for the training/evaluation code and the audio manipulation scripts used for the paper.

Model description

The model is NbAiLab/nb-whisper-medium's encoder with a classification head on top (WhisperForAudioClassification), fully fine-tuned (no layers frozen) for sequence classification over 4 dialect labels. Unlike the unmodified-audio model, this model was trained and evaluated on speech whose F0 (pitch) contour was monotonized — flattened to the utterance's average F0. This removes pitch-based prosodic variation while keeping segmental/phonetic content intact, isolating the contribution of non-pitch cues (timing, energy, phonetic content) to dialect classification.

Intended uses & limitations

Intended for research on Norwegian dialect identification, specifically on studying how much dialect information survives when pitch-based prosody is removed. Audio fed to this model should be F0-monotonized the same way as during training (see the did_prosody_whisper repo) — running unmodified audio through this model does not reproduce the paper's "unmodified" condition results; use scribe-project/nb-whisper-dialect-id-4dialect for that. It is not intended for consequential decisions about individuals (e.g. hiring, legal, or identity-verification contexts).

Known limitations:

• Trained only on adult parliamentary speech; accuracy on other domains (conversational speech, children, non-native speakers) is not guaranteed.
• This repo ships the checkpoint from the final training epoch (epoch 3), for consistency with the unmodified and low-pass models in this series, rather than the epoch-2 checkpoint, which had marginally higher eval accuracy (0.8733 vs 0.8727 — see Training results below).
• The 4 dialect categories are coarse regional groupings, not a fine-grained dialect taxonomy, and boundaries between adjacent regions are inherently fuzzy.

How to use

import torch
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
import librosa

model_id = "scribe-project/nb-whisper-dialect-id-4dialect-monotonize"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModelForAudioClassification.from_pretrained(model_id)

# audio should be F0-monotonized the same way as during training — see the
# did_prosody_whisper repo for the monotonization procedure
audio, sr = librosa.load("path/to/monotonized_audio.wav", sr=feature_extractor.sampling_rate, mono=True)

inputs = feature_extractor(
    audio,
    sampling_rate=feature_extractor.sampling_rate,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = torch.argmax(logits, dim=-1).item()
print(model.config.id2label[predicted_id])

Audio should be mono, resampled to 16kHz. Clips longer than 30 seconds were randomly subsampled to 30 seconds during training.

Training and evaluation data

Trained on the SSC (Storting/Parliament Speech Corpus), using a speaker-disjoint train/validation split (no speaker overlap between train and eval). Each example is a single-speaker audio segment labeled with one of the 4 dialect categories above. Prior to training, each utterance's F0 contour was monotonized to its average F0 (estimated with REAPER); see the did_prosody_whisper repo for the exact Praat scripts used.

Training procedure

Training hyperparameters

• learning_rate: 3e-05
• train_batch_size: 16 (per device)
• eval_batch_size: 32 (per device)
• seed: 0
• distributed_type: multi-GPU
• num_devices: 6
• gradient_accumulation_steps: 2
• total_train_batch_size: 192
• optimizer: Adam (betas=(0.9, 0.999), epsilon=1e-08)
• lr_scheduler_type: linear, warmup_ratio 0.1
• num_epochs: 3.0
• mixed_precision_training: Native AMP
• feature encoder: not frozen (full fine-tune)

Training results

The weights published in this repo are from epoch 3 (checkpoint-1356), the final training checkpoint — not the epoch-2 checkpoint, which had marginally higher eval accuracy but is inconsistent with the checkpoint selection used for the unmodified and low-pass conditions above.

Framework versions

• Transformers 4.40.0
• PyTorch 2.2.2+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1