whisper-large-v3-arabic-dialect-30s

Whisper-large-v3 Arabic Dialect Identification (30s)

Model Description

This model identifies 17 regional Arabic dialects from speech recordings of up to 30 seconds. It is built on top of OpenAI's Whisper-large-v3 encoder, fine-tuned on the ADI17 dataset — a large-scale Arabic dialect corpus sourced from YouTube broadcasts covering 17 Arab countries.

The system takes up to 30 seconds of raw audio, converts it into an 80-channel log-mel spectrogram, passes it through the fine-tuned Whisper encoder, and produces a probability distribution over 17 dialect classes using a custom attention pooling mechanism and a multi-layer classification head.

Compared to the 15s version, this model uses longer audio windows during training, giving the encoder more speech context per sample and achieving higher accuracy (95.11% vs 92.58%).

---

Supported Dialects

---

Architecture

Input Audio (up to 30s)
        ↓
WhisperFeatureExtractor → 80-channel log-mel spectrogram (80 × 3000)
        ↓
Whisper-large-v3 Encoder (32 Transformer layers)
  ├── Layers 1–24 : Frozen (low-level acoustic features preserved)
  └── Layers 25–32: Fine-tuned (high-level dialect-specific features)
        ↓
Attention Pooling → learned weighted average over 1500 time frames → 1280-dim vector
        ↓
MLP Classifier Head:
  LayerNorm → Dropout(0.15)
  → Linear(1280 → 512) → GELU
  → Dropout(0.15)
  → Linear(512 → 256) → GELU
  → Dropout(0.075)
  → Linear(256 → 17) → dialect probabilities

---

Training Details

---

Results

Per-Dialect F1 Scores

Best dialect: Lebanese (LEB) at 97.81% F1 Hardest dialect: Libyan (LIB) at 87.16% F1

Comparison with 15s Model

The 30s model consistently outperforms the 15s model across all dialects, as longer audio windows provide more phonetic, prosodic, and lexical context for dialect identification.

---

Data Preprocessing Pipeline

Raw .wav audio files were preprocessed as follows:

1. Converted to mono and resampled to 16 kHz
2. Segmented into non-overlapping 30-second windows (zero-padded if shorter)
3. Maximum 10 windows per recording to prevent long files from dominating
4. Each window converted to an 80-channel log-mel spectrogram using WhisperFeatureExtractor
5. Features saved as float16 in compressed HDF5 files for fast training I/O

Training set was stratified and capped at 8,000 samples per class.

---

How to Use

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import librosa
from transformers import WhisperModel, WhisperFeatureExtractor
from huggingface_hub import snapshot_download

# Download model
model_dir = snapshot_download("mesoj8/whisper-large-v3-arabic-dialect-30s")

# Load feature extractor
feature_extractor = WhisperFeatureExtractor.from_pretrained(model_dir)

# Define architecture (must match training)
class AttentionPool(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.attn = nn.Linear(d_model, 1, bias=False)
    def forward(self, x):
        weights = F.softmax(self.attn(x), dim=1)
        return (x * weights).sum(dim=1)

class WhisperDialectClassifier(nn.Module):
    def __init__(self, whisper_model, num_classes=17, dropout=0.15):
        super().__init__()
        self.whisper = whisper_model
        hs = self.whisper.config.d_model
        self.pool = AttentionPool(hs)
        self.classifier = nn.Sequential(
            nn.LayerNorm(hs), nn.Dropout(dropout),
            nn.Linear(hs, 512), nn.GELU(), nn.Dropout(dropout),
            nn.Linear(512, 256), nn.GELU(), nn.Dropout(dropout / 2),
            nn.Linear(256, num_classes)
        )
    def forward(self, x):
        h = self.whisper.encoder(input_features=x).last_hidden_state
        return self.classifier(self.pool(h))

# Load model
device = "cpu"
whisper_base = WhisperModel.from_pretrained(model_dir)
model = WhisperDialectClassifier(whisper_base).to(device)
model.pool.load_state_dict(torch.load(f"{model_dir}/pool.pth", map_location=device, weights_only=True))
model.classifier.load_state_dict(torch.load(f"{model_dir}/classifier.pth", map_location=device, weights_only=True))
model.eval()

# Inference
audio, sr = librosa.load("your_audio.wav", sr=16000, mono=True)
target = 30 * 16000  # 480000 samples
audio = np.pad(audio, (0, max(0, target - len(audio))))[:target]

inputs = feature_extractor(audio, sampling_rate=16000, return_tensors="pt")
with torch.inference_mode():
    probs = torch.softmax(model(inputs["input_features"]), dim=-1)[0]

ID2LABEL = {0:"ALG",1:"EGY",2:"IRA",3:"JOR",4:"KSA",5:"KUW",6:"LEB",7:"LIB",
            8:"MAU",9:"MOR",10:"OMA",11:"PAL",12:"QAT",13:"SUD",14:"SYR",15:"UAE",16:"YEM"}
print(f"Predicted dialect: {ID2LABEL[probs.argmax().item()]} ({probs.max().item()*100:.1f}%)")

---

Limitations

• Libyan (LIB) and Syrian (SYR) dialect performance is lower due to acoustic overlap with neighboring dialects
• The model was trained on broadcast and YouTube speech — performance may degrade on conversational, noisy, or phone-quality audio
• Code-switching (mixing Arabic with other languages) may reduce accuracy
• Audio shorter than 3 seconds is not supported
• Only the Whisper encoder is used — the decoder (transcription) is not activated

---

Related Models

---

Citation

@misc{mesoj8_arabic_dialect_30s_2026,
  author    = {Meso},
  title     = {Whisper-large-v3 Fine-tuned for Arabic Dialect Identification (ADI17, 30s windows)},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/mesoj8/whisper-large-v3-arabic-dialect-30s}
}