Downloads · 30 days
0
tokenmonster
alasdairforsythe/tokenmonster
tokenmonster is a machine learning model from alasdairforsythe. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The documentation and code is available on Github alasdairforsythe/tokenmonster.
Access
Public
Updated Sep 5, 2023
Repo size
1.1 GB
Likes
19
Public
Repo makeup
Click a slice to open those files.
.vocab387 MB · 85%
At a glance
- License
- mit
- Access
- Public
- Created
- May 31, 2023
- Updated
- Sep 5, 2023
- SHA
- c15c899a
Model card
From the Hugging Face model README
TokenMonster
The documentation and code is available on Github alasdairforsythe/tokenmonster.
The pretrained vocabularies are all available for download here.
July 11: TokenMonster v1.1.1 has been released. The "420" prebuilt vocabularies are being released as they are completed, at a rate of around 10 per day.
Choose a dataset from:
codeenglishenglishcodefiction
Choose a vocab size from:
1024204840968000160002400032000400005025665536100256
Choose an optimization mode from:
unfilteredcleanbalancedconsistentstrict
For a capcode disabled vocabulary add:
nocapcode
And finally add the version number:
v1
Examples:
fiction-24000-consistent-v1code-4096-clean-nocapcode-v1
There are two additional vocabularies:
gpt2llama
- License
- mit
- Created
- May 31, 2023
- Updated
- Sep 5, 2023