<div align="center">
<h1>Indonesian Automatic Speech Recognition with XLSR-53</h1>
<p>A fine-tuned model for Automatic Speech Recognition (ASR) in Indonesian, achieving competitive performance with a significantly reduced Word Error Rate (WER) using a KenLM language model.</p>
</div>
<p align="center">
<a href="#how-to-use"><strong>How to Use</strong></a> ·
<a href="#evaluation-and-results"><strong>Evaluation Results</strong></a> ·
<a href="#citation"><strong>Citation</strong></a> ·
<a href="https://huggingface.co/spaces/panjiariputra/indonesian-speech-recognition" target="_blank"><strong>Try on Spaces</strong></a> ·
<a href="https://doi.org/10.18280/isi.270614" target="_blank"><strong>Read the Paper</strong></a>
</p>
<br>
This repository contains the official fine-tuned model from the research paper "Indonesian Automatic Speech Recognition with XLSR-53". The study focuses on developing a robust Indonesian ASR system by fine-tuning the pre-trained cross-lingual XLSR-53 (<strong><a href="https://huggingface.co/facebook/wav2vec2-large-xlsr-53">facebook/wav2vec2-large-xlsr-53</a></strong>) model.
The key contribution of this work is demonstrating that a competitive Word Error Rate (WER) can be achieved with a relatively small dataset (24 hours). The model's accuracy is significantly boosted by integrating a 4-gram KenLM language model, which successfully reduces the WER from 20% to 12% on the Common Voice test set.
<p align="center">
<img alt="Proposed Methodology" src="https://cdn-uploads.huggingface.co/production/uploads/6185e35858cb1f8c362f943c/iiGX4hokk2SD1yySLiFWd.png" width="800">
<br>
<em>Proposed Methodology </em>
</p>
Model Details
<ul>
<li><strong>Base Model:</strong> This model is built upon the wav2vec 2.0 architecture, specifically the <strong>XLSR-53</strong> pre-trained model (<code>facebook/wav2vec2-large-xlsr-53</code>) which was trained on 53 languages.</li>
<li><strong>Task:</strong> Automatic Speech Recognition (ASR).</li>
<li><strong>Language:</strong> Indonesian (id).</li>
<li><strong>Library:</strong> Transformers.</li>
<li><strong>Framework:</strong> The approach involves fine-tuning the pre-trained model using a Connectionist Temporal Classification (CTC) loss function.</li>
</ul>
Authors
<ul>
<li>Panji Arisaputra</li>
<li>Amalia Zahra</li>
</ul>
<p><em>Computer Science Department, BINUS Graduate Program, Bina Nusantara University, Jakarta, Indonesia.</em></p>
Datasets Used for Training
<p>
A total of three speech datasets were combined to fine-tune the model, and an additional large text corpus was used to build the language model.
</p>
<h4>Speech Data for Fine-Tuning:</h4>
<p>The total combined duration of speech data is <strong>24 hours, 18 minutes, and 1 second</strong>.</p>
<ul>
<li><strong>TITML-IDN:</strong> A clean speech corpus containing <strong>14.5 hours</strong> of audio from 20 speakers reading phonetically balanced sentences.</li>
<li><strong>Magic Data:</strong> A <strong>3.5-hour</strong> corpus of scripted daily-use sentences from 10 speakers, recorded in various environments.</li>
<li><strong>Common Voice (Indonesian):</strong> A crowdsourced dataset containing <strong>~6.2 hours</strong> of speech from 170 speakers in diverse, non-clean environments.</li>
</ul>
<h4>Text Data for Language Model:</h4>
<ul>
<li>In addition to the transcripts from the speech datasets, the <strong>OSCAR corpus</strong> (unshuffled_deduplicated_id subset) was used to build the KenLM language model. To ensure a balanced vocabulary, only 6% of its 2.3 billion Indonesian words were included.</li>
</ul>
<h4>Data Preprocessing</h4>
<p>The datasets underwent a standardized preprocessing pipeline:</p>
<ol>
<li><strong>Data Splitting:</strong> Datasets were split into training (90%) and validation (10%) sets.</li>
<li><strong>Audio Standardization:</strong> All audio files were converted to WAV format with a single channel and resampled to a 16 kHz sampling rate to match the pre-trained model's requirements.</li>
<li><strong>Text Normalization:</strong> Transcriptions were cleaned by removing special characters and converting all text to lowercase to create a unified vocabulary.</li>
</ol>
<h3 id="evaluation-and-results">Evaluation and Results</h3>
<p>
The model was evaluated against a similar model from a previous study by Syahputra & Zahra (2021), using the <strong>Word Error Rate (WER)</strong> metric. The evaluation on the Common Voice test split serves as the primary benchmark.
</p>
<p>
The results show that this XLSR-53 model outperforms the previous wav2vec 2.0-based model. The integration of a <strong>4-gram KenLM language model</strong> was crucial, providing an <strong>8% absolute reduction in WER</strong> (from 20% down to 12%).
</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Data Training & Validation</th>
<th>Language Model</th>
<th>Test Set</th>
<th>WER (%)</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2"><b>This Study (XLSR-53)</b></td>
<td rowspan="2">TITML-IDN + Magic Data + Common Voice (24h 18m)</td>
<td>—</td>
<td>Common Voice</td>
<td>20.306%</td>
</tr>
<tr>
<td><b>4-gram KenLM</b></td>
<td>Common Voice</td>
<td><b>12,213%</b></td>
</tr>
<tr>
<td rowspan="2">Benchmark (Syahputra & Zahra, 2021)</td>
<td rowspan="2">BahasaKita batch 10 – 12 (75h)</td>
<td>—</td>
<td>Common Voice</td>
<td>21.000%</td>
</tr>
<tr>
<td>3-gram KenLM</td>
<td>Common Voice</td>
<td>41.000%</td>
</tr>
</tbody>
</table>
<p style="font-size: smaller; text-align: center; margin-top: 5px;">*WER results extracted from Table 3 of the research paper. The benchmark model's high WER with LM is noted in the paper.</p>
<h3 id="how-to-use">How to Use</h3>
<p>
You can use this model with the <code>transformers</code> library pipeline. For optimal performance, as demonstrated in the research paper, we strongly recommend integrating the provided <strong>4-gram KenLM language model</strong>.
</p>
<pre><code>pip install transformers torch torchaudio librosa
# For decoding with the language model:
pip install pyctcdecode==0.4.0 kenlm
</code></pre>
<h4>Without Language Model</h4>
<pre><code class="language-python">from transformers import AutoProcessor, AutoModelForCTC, pipeline
import torch
import librosa
# Load processor and model
processor = AutoProcessor.from_pretrained("panjiarisaputra/indonesian-asr-xlsr-53")
model = AutoModelForCTC.from_pretrained("panjiarisaputra/indonesian-asr-xlsr-53")
# Initialize ASR pipeline
asr_pipeline = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
device=0 if torch.cuda.is_available() else -1
)
# Load an audio file (must be 16kHz, mono)
audio_path = "path/to/your/audio.wav"
speech_array, sampling_rate = librosa.load(audio_path, sr=16000)
# Run transcription
transcription = asr_pipeline(speech_array)
print(transcription)
# Output: {'text': '...transcribed text...'}
</code></pre>
<h4 id="with-language-model">With Language Model Integration (Recommended)</h4>
<p>
For the best accuracy and lowest Word Error Rate (WER), use <code>pyctcdecode</code> with the 4-gram KenLM model (e.g., <code>4gram.arpa</code>) created during the research.
</p>
<pre><code class="language-python">from transformers import AutoProcessor, AutoModelForCTC
from pyctcdecode import build_ctcdecoder
import torch
import librosa
# Load processor and model
processor = AutoProcessor.from_pretrained("panjiariputra/indonesian-xlsr_53-LARGE-4gram")
model = AutoModelForCTC.from_pretrained("panjiariputra/indonesian-xlsr_53-LARGE-4gram")
# Get vocabulary and build the decoder with the language model
vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k: v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}
decoder = build_ctcdecoder(
labels=list(sorted_vocab_dict.keys()),
kenlm_model_path="path/to/your/4gram.arpa" # Path to your KenLM model
)
# Load audio (16kHz, mono)
audio_path = "path/to/your/audio.wav"
speech_array, _ = librosa.load(audio_path, sr=16000)
# Get model logits
with torch.no_grad():
inputs = processor(speech_array, sampling_rate=16000, return_tensors="pt", padding=True)
logits = model(**inputs).logits.cpu().numpy()[0]
# Decode using KenLM
lm_transcription = decoder.decode(logits)
print({"text": lm_transcription})
# Output: {'text': '...more accurate transcribed text...'}
</code></pre>
<h3 id="citation">Publication and Citation</h3>
<p>This work was published in <strong>Ingénierie des Systèmes d'Information, Vol. 27, No. 6, December, 2022</strong>. You can download the full paper <a href="https://doi.org/10.18280/isi.270614" target="_blank">here</a>. If you use this model or the findings from the paper in your research, please cite:</p>
<pre><code>@article{Arisaputra2022XLSR53,
author = {Panji Arisaputra and Amalia Zahra},
title = {Indonesian Automatic Speech Recognition with XLSR-53},
journal = {Ingénierie des Systèmes d'Information},
volume = {27},
number = {6},
pages = {973--982},
year = {2022},
doi = {10.18280/isi.270614}
}
</code></pre>