Downloads · 30 days
18
12% of all-time downloads
aaqibsaeed/plixkws
plixkws is a machine learning model from aaqibsaeed. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
As technology advances and digital devices become prevalent, seamless human-machine communication is increasingly gaining significance. The growing adoption of mobile, wearable, and other Internet of Things (IoT) devi…
Downloads · 30 days
18
12% of all-time downloads
All-time downloads
149
Public
Repo size
496 MB
Likes
2
Public
Click a slice to open those files.
.pt496 MB · 100%
From the Hugging Face model README
As technology advances and digital devices become prevalent, seamless human-machine communication is increasingly gaining significance. The growing adoption of mobile, wearable, and other Internet of Things (IoT) devices has changed how we interact with these smart devices, making accurate spoken words recognition a crucial component for effective interaction. However, building robust spoken words detection system that can handle novel keywords remains challenging, especially for low-resource languages with limited training data. Here, we propose PLiX, a multilingual and plug-and-play keyword spotting system that leverages few-shot learning to harness massive real-world data and enable the recognition of unseen spoken words at test-time. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages, achieving state-of-the-art performance while being highly efficient. Extensive evaluations show that PLiX can generalize to novel spoken words given as few as just one support example and performs well on unseen languages out of the box. We release models and inference code to serve as a foundation for future research and voice-enabled user interface development for emerging devices.
We provide the library for our PLiX model:
pip install plixkws
Then you can follow the below usage or refer to test_model.py.
import torch
from plixkws import model, util
support_examples = ["./test_clips/aandachtig.wav", "./test_clips/stroom.wav",
"./test_clips/persbericht.wav", "./test_clips/klinkers.wav",
"./test_clips/zinsbouw.wav"]
classes = ["aandachtig", "stroom", "persbericht", "klinkers", "zinsbouw"]
int_indices = [0,1,2,3,4]
fws_model = model.load(encoder_name="base", language="en", device="cpu")
support = {
"paths": support_examples,
"classes": classes,
"labels": torch.tensor(int_indices),
}
support["audio"] = torch.stack([util.load_clip(path) for path in support["paths"]])
support = util.batch_device(support, device="cpu")
query = {
"paths": ["./test_clips/query_klinkers.wav", "./test_clips/query_stroom.wav"]
}
query["audio"] = torch.stack([util.load_clip(path) for path in query["paths"]])
query = util.batch_device(query, device="cpu")
with torch.no_grad():
predictions = fws_model(support, query)
# !pip install pyaudio
import numpy as np
import pyaudio
import torch
from plixkws import model, util
sample_rate = 16000
frames_per_buffer = 512
support_examples = ["./test_clips/aandachtig.wav", "./test_clips/stroom.wav",
"./test_clips/persbericht.wav", "./test_clips/klinkers.wav",
"./test_clips/zinsbouw.wav"]
classes = ["aandachtig", "stroom", "persbericht", "klinkers", "zinsbouw"]
int_indices = [0,1,2,3,4]
support = {
"paths": support_examples,
"classes": classes,
"labels": torch.tensor(int_indices)
}
support["audio"] = torch.stack([util.load_clip(path) for path in support["paths"]])
support = util.batch_device(support, device="cpu")
fws_model = model.load(encoder_name="small", language="nl", device="cpu")
p = pyaudio.PyAudio()
stream = p.open(format = pyaudio.paInt16, channels=1,
rate=sample_rate, input=True, frames_per_buffer=frames_per_buffer)
frames = []
while True:
data = stream.read(frames_per_buffer)
buffer = np.frombuffer(data, dtype=np.int16)
frames.append(buffer)
if len(frames) * frames_per_buffer / sample_rate >= 1:
audio = np.concatenate(frames)
audio = audio.astype(float) / np.iinfo(np.int16).max
query = {"audio":torch.tensor(audio[np.newaxis, np.newaxis,:], dtype=torch.float32)}
query = util.batch_device(query, device="cpu")
with torch.no_grad():
predictions = fws_model(support, query)
print(classes[predictions.item()])
frames = []
| Language | Encoder Name |
|---|---|
| Multilingual | base_multi |
| Multilingual | small_multi |
| English | base_en |
| Arabic | small_ar |
| Czech | small_cs |
| German | small_de |
| Greek | small_el |
| English | small_en |
| Estonian | small_et |
| Spanish | small_es |
| Persian | small_fa |
| French | small_fr |
| Indonesian | small_id |
| Italian | small_it |
| Kyrgyz | small_ky |
| Dutch | small_nl |
| Polish | small_pl |
| Portuguese | small_pt |
| Russian | small_ru |
| Kinyarwanda | small_rw |
| Swedish | small_sv-SE |
| Turkish | small_tr |
| Tatar | small_tt |
If you find this work useful, please cite our paper:
@article{saeed2023plix,
title={Plug-and-Play Multilingual Few-shot Spoken Words Recognition},
author={Saeed, Aaqib and Tsouvalas, Vasileios},
journal={arXiv preprint arXiv:2305.03058},
year={2023}
}