wav2vec2-large-mms-1b-cv_vp

Model Card for GaborMadarasz/wav2vec2-large-mms-1b-cv_vp

This model is a Hungarian Automatic Speech Recognition (ASR) system fine-tuned from the facebook/mms-1b-all base model. It utilizes the Wav2Vec2 architecture with a Connectionist Temporal Classification (CTC) head, optimized specifically for Hungarian speech.

Key Metrics:

• Hungarian FLEURS Test WER: 0.3068
• Hungarian FLEURS Test CER: 0.0727
• Common Voice + Voxpopuli (500 + 500 held-out samples) WER: 0.1275

Model Details

Model Description

• Developed by: Gabor Madarasz
• Model type: Wav2Vec2ForCTC (Connectionist Temporal Classification)
• Language(s): Hungarian (hu)
• License: CC0-1.0
• Finetuned from model: facebook/mms-1b-all

Model Sources

• Repository: https://huggingface.co/GaborMadarasz/wav2vec2-large-mms-1b-hungarian

Uses

Direct Use

This model is intended for direct use in Hungarian speech-to-text (STT) applications. It can transcribe clear, single-speaker Hungarian audio (e.g., podcasts, audiobooks, recorded speech) into text with high character-level accuracy.

Downstream Use

The model serves as an excellent acoustic baseline for downstream applications. For production environments, it is highly recommended to integrate this model with an n-gram Language Model (e.g., KenLM) via beam search decoding (e.g., pyctcdecode) to resolve phonetically similar word errors and further reduce the Word Error Rate (WER).

Out-of-Scope Use

• Transcribing audio with heavy background noise, overlapping speakers, or extreme reverberation.
• Processing non-Hungarian languages.
• Real-time streaming ASR without appropriate chunking/VAD modifications.

Bias, Risks, and Limitations

Limitations

• CTC Characteristics: As a CTC model, it may occasionally produce phonetically similar word substitutions (e.g., homophones) or struggle with precise punctuation placement. The low CER compared to WER confirms strong phonetic accuracy, making LM integration highly effective.
• Domain Shift: Performance may degrade on highly specialized technical jargon, heavy regional dialects, or code-switching not represented in the training data.
• Demographic Bias: The training data (e.g., Common Voice) may overrepresent certain demographics, potentially leading to slightly higher error rates for underrepresented accents or speech patterns.

Recommendations

Users should be aware of these limitations. For critical applications, implement a post-processing step or a Language Model decoder to correct contextually unlikely word sequences. Always validate the model's performance on your specific target domain before deployment.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from datasets import load_dataset

# Load model and processor
model_id = "GaborMadarasz/wav2vec2-large-mms-1b-cv_vp"
processor = Wav2Vec2Processor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id, target_lang="hun")

# Load dummy audio (replace with your own Hungarian audio file)
# Ensure the audio is resampled to 16,000 Hz
dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_array = dataset[0]["audio"]["array"]
sampling_rate = dataset[0]["audio"]["sampling_rate"]

# Process audio and predict
inputs = processor(audio_array, sampling_rate=sampling_rate, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(inputs.input_values).logits

# Decode output
pred_ids = torch.argmax(logits, dim=-1)
predicted_text = processor.batch_decode(pred_ids)[0]

print(f"Prediction: {predicted_text}")

Training Details

Training Data

The model was trained on:

1. Mozilla Common Voice 24.0 (Hungarian): Primary open-source, crowdsourced speech data.
2. Facebook Voxpopuli (Hungarian subset)

Training Procedure

The model was fine-tuned using the Hugging Face transformers Trainer API.

Training Hyperparameters

• Learning Rate: 8e-4 (with linear scheduler)
• Batch Size: 2 per device
• Gradient Accumulation Steps: 16 (Effective batch size: 32)
• Mixed Precision: BF16
• Gradient Checkpointing: Enabled (to fit within 12GB VRAM constraints)
• Epochs: 5

Speeds, Sizes, Times

• Checkpoint Size: ~3.8 GB
• Hardware: NVIDIA RTX3060 GPU with 12GB VRAM

Evaluation

Testing Data, Factors & Metrics

• Testing Data: Hungarian FLEURS test set, and a 1000-sample held-out subset of Common Voice + Voxpopuli.
• Metrics:
  • Word Error Rate (WER): Standard metric for ASR, measuring word-level accuracy. 3 - Character Error Rate (CER): Crucial for agglutinative languages like Hungarian, as it accurately captures phonetic and morphological errors (e.g., missing or incorrect suffixes) that WER might over-penalize.

Results

Model Card Contact

For questions, issues, or collaboration, please contact: [email protected]