nb-whisper-dialect-id-4dialect

nb-whisper-dialect-id-4dialect

A NbAiLab/nb-whisper-medium encoder fine-tuned for 4-class Norwegian dialect identification on unmodified (natural) speech.

Given an audio clip of Norwegian speech, the model predicts which of four broad dialect regions the speaker belongs to:

This model is one of three "prosody-condition" models (unmodified / low-pass / monotonize) trained for:

Phoebe Parsons, Heming Strømholt Bremnes, Knut Kvale, Torbjørn Svendsen, and Giampiero Salvi. (2025). Effects of Prosodic Information on Dialect Classification Using Whisper Features. In Proceedings of Interspeech 2025, pages 2785–2789. doi: 10.21437/Interspeech.2025-200

The other two conditions are trained on the same data with the speech signal manipulated to isolate or remove prosodic (F0) cues:

• scribe-project/nb-whisper-dialect-id-4dialect-low-pass — low-pass filtered audio
• scribe-project/nb-whisper-dialect-id-4dialect-monotonize — F0-monotonized audio

See the did_prosody_whisper repo for the training/evaluation code used for the paper.

Model description

The model is NbAiLab/nb-whisper-medium's encoder with a classification head on top (WhisperForAudioClassification), fully fine-tuned (no layers frozen) for sequence classification over 4 dialect labels.

Intended uses & limitations

Intended for research on Norwegian dialect identification, and specifically as the "unmodified audio" baseline against which the low-pass and monotonize conditions are compared to study the role of prosody in dialect classification. It is not intended for consequential decisions about individuals (e.g. hiring, legal, or identity-verification contexts).

Known limitations:

• Trained only on adult parliamentary speech; accuracy on other domains (conversational speech, children, non-native speakers) is not guaranteed.
• This repo ships the checkpoint from the final training epoch (epoch 3), for consistency with the low-pass and monotonize models in this series, rather than the checkpoint with the highest eval accuracy (epoch 1 reached 0.9318 before accuracy dropped in later epochs — see Training results below).
• The 4 dialect categories are coarse regional groupings, not a fine-grained dialect taxonomy, and boundaries between adjacent regions are inherently fuzzy.

How to use

import torch
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
import librosa

model_id = "scribe-project/nb-whisper-dialect-id-4dialect"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModelForAudioClassification.from_pretrained(model_id)

audio, sr = librosa.load("path/to/audio.wav", sr=feature_extractor.sampling_rate, mono=True)

inputs = feature_extractor(
    audio,
    sampling_rate=feature_extractor.sampling_rate,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = torch.argmax(logits, dim=-1).item()
print(model.config.id2label[predicted_id])

Audio should be mono, resampled to 16kHz. Clips longer than 30 seconds were randomly subsampled to 30 seconds during training.

Training and evaluation data

Trained on the SSC (Storting/Parliament Speech Corpus), using a speaker-disjoint train/validation split (no speaker overlap between train and eval). Each example is a single-speaker audio segment labeled with one of the 4 dialect categories above. Audio was used unmodified (no low-pass filtering or monotonization).

Training procedure

Training hyperparameters

• learning_rate: 3e-05
• train_batch_size: 16 (per device)
• eval_batch_size: 32 (per device)
• seed: 0
• distributed_type: multi-GPU
• num_devices: 6
• gradient_accumulation_steps: 2
• total_train_batch_size: 192
• optimizer: Adam (betas=(0.9, 0.999), epsilon=1e-08)
• lr_scheduler_type: linear, warmup_ratio 0.1
• num_epochs: 3.0
• mixed_precision_training: Native AMP
• feature encoder: not frozen (full fine-tune)

Training results

The weights published in this repo are from epoch 3 (checkpoint-1359), the final training checkpoint — not the epoch-1 checkpoint, which had the highest single-epoch eval accuracy but is inconsistent with the checkpoint selection used for the low-pass and monotonize conditions above.

Framework versions

• Transformers 4.40.0
• PyTorch 2.2.2+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1