Zipformer-30M-RNNT-Streaming-6000h

Vietnamese Streaming Speech-to-Text (ASR) — ZipFormer-30M-RNNT-Streaming-6000h

🔍 Overview

The Vietnamese Streaming Speech-to-Text (ASR) model is built on the ZipFormer architecture with chunk size 16,32,64 — an improved variant of the Conformer — featuring only 30 million parameters yet.
On CPU, the model can transcribe a 1-second audio chunk in just 0.05 seconds, designed for streaming-based tasks with low latency requirements.

---

🚀 Online Demo

You can test the streaming and none-streaming model directly here:
👉 https://huggingface.co/spaces/hynt/k2-automatic-speech-recognition-demo

---

⚙️ Model Architecture and Training strategy:

• Architecture: ZipFormer
• Parameters: ~30M
• Language: Vietnamese
• Loss Function: RNN-Transducer (RNNT Loss)
• Chunk Size: 16, 32, 64
• Framework: PyTorch + k2
• Training strategy: Carefully preprocess the data, apply an augmentation strategy based on the distribution of out-of-vocabulary (OOV) tokens and refine the transcriptions using Whisper.
• Optimized for: High-speed CPU inference

---

🧠 Training Data

The model was trained on approximately 6000 hours of high-quality Vietnamese speech collected from various public datasets:

---

🧪 Evaluation Results

Lower is better (WER %)

---

🏆 Achievements

By training this none-streaming model architecture on 4,000 hours of data, I won First Place in the Vietnamese Language Speech Processing (VLSP) competition 2025. Comprehensive details about training data, optimization strategies, architecture improvements, and evaluation methodologies are available in the paper below:

👉 Read the full paper on ACL

---

⚡ Inference Speed

---

⚙️ How to Run This Model

Please refer to the following guides for instructions on how to run and deploy this model:

• For Torch JIT Script: https://k2-fsa.github.io/sherpa/
• For ONNX: https://k2-fsa.github.io/sherpa/onnx/
• For Streaming Web Test:: https://github.com/k2-fsa/sherpa/tree/master/sherpa/bin

💬 Summary

The ZipFormer-30M-RNNT-6000h and ZipFormer-30M-RNNT-Streaming-6000h model demonstrates that a lightweight architecture can still achieve state-of-the-art accuracy for Vietnamese ASR.
It is designed for fast deployment on CPU-based systems, making it ideal for real-time speech recognition, callbots, and embedded speech interfaces.

---