Downloads · 30 days
0
wenet_speech_lm
pkufool/wenet_speech_lm
wenet_speech_lm is a machine learning model from pkufool. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Access
Public
Updated Nov 24, 2021
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
Other1.2 KB · 69%
At a glance
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Nov 24, 2021
- SHA
- db634d7f
Model card
From the Hugging Face model README
- Install requirements
pip install jieba
- Generate words.txt
data_dir=/path/to/wenetspeech
# the data_dir contains:
# tree -L 2 .
# .
# |-- TERMS_OF_ACCESS
# |-- WenetSpeech.json
# |-- audio
# |-- dev
# |-- test_meeting
# |-- test_net
# `-- train
grep "\"text\":" $data_dir/WenetSpeech.json | sed -e 's/["text: ]*//g' > text.txt
python -m jieba -d " " text.txt > tokenized.txt
cat tokenized.txt | awk '{for(i=1;i<=NF;i++)print $i}' | sort | uniq > words.txt
- Generate N-gram model
- Created
- Mar 2, 2022
- Updated
- Nov 24, 2021