Downloads · 30 days
0
tokenizer_g2pen_v2
therealvul/tokenizer_g2pen_v2
tokenizer_g2pen_v2 is a machine learning model from therealvul. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This tokenizer was trained on a small corpus of concatenated ARPAbet pronunciation tokens + punctuation from the python g2pen library computed over the entire synthbot/pony-speech dataset and 240k lines from genericsk…
Access
Public
Updated Sep 17, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json41.1 KB · 95%
At a glance
- Access
- Public
- Created
- Sep 17, 2024
- Updated
- Sep 17, 2024
- SHA
- 245c2d80
Model card
From the Hugging Face model README
This tokenizer was trained on a small corpus of concatenated ARPAbet pronunciation tokens + punctuation from the python g2p_en library computed over the entire synthbot/pony-speech dataset and 240k lines from generics_kb_best, from community-datasets/generics_kb.
i.e. But one on one, let's clean it. -> BAH1T WAH1N AA1N WAH1N , LEH1TS KLIY1N IH1T . Uses BPE with vocab size of 1024.
It is trained on the same data as https://huggingface.co/therealvul/tokenizer_g2pen with the following differences:
- It does not split on whitespace as a token
- It uses ByteLevel in pretokenization/decoding step
- It uses a token vocab size of 1024 instead of 384
- Languages
- en
- Created
- Sep 17, 2024
- Updated
- Sep 17, 2024