SynTTS-Commands-Media-Benchmarks

SynTTS Commands Media Benchmarks

<!-- Badges Row -->

---

🚀 Project Navigation

Welcome to the official model repository for the paper "SynTTS-Commands". Here you can find the pre-trained checkpoints for KWS tasks.

• 📄 Paper: Read the detailed technical report on arXiv.
• 💾 Dataset: Download the training data at SynTTS-Commands-Media-Dataset.
• 💻 Code: Access training scripts and inference code on GitHub.

---

---

📈 Benchmark Results and Analysis

We present a comprehensive benchmark of six representative acoustic models on the SynTTS-Commands-Media Dataset across both English (EN) and Chinese (ZH) subsets. All models are evaluated in terms of classification accuracy, cross-entropy loss, and parameter count, providing insights into the trade-offs between performance and model complexity in multilingual voice command recognition.

Performance Summary

🔍 Key Findings

Our results demonstrate that the SynTTS-Commands dataset supports high-accuracy command recognition in both languages. Notably, the top-performing models achieve over 99.4% accuracy on English and nearly 98% on Chinese, confirming the dataset’s quality and suitability for real-world deployment.

• Top Performers: Among all models, CRNN attains the best English accuracy (99.50%) and the lowest loss (0.0163). MobileNet-V1 yields the lowest loss on Chinese (0.0552) and competitive English performance (matching CRNN’s 99.50% accuracy). Interestingly, EfficientNet shows slightly higher Chinese accuracy (97.93%) than MobileNet-V1, suggesting better calibration or robustness despite a higher loss.

• Accuracy-Complexity Trade-off: Lightweight models exhibit a clear trade-off. MicroCNN, with only ~4.2K parameters, achieves 93.22% accuracy on English but drops to 80.14% on Chinese, highlighting the increased difficulty of modeling tonal and phonetic richness in Mandarin with ultra-compact architectures. DS-CNN and TC-ResNet, with under 70K parameters, already recover strong performance (>96.5% in both languages), underscoring their efficiency for resource-constrained applications.

Overall, the benchmark establishes strong baselines across a wide spectrum of model scales—from ultra-light MicroCNN to modern EfficientNet—demonstrating that moderate-complexity models can deliver near-SOTA performance suitable for edge deployment.