Downloads · 30 days
0
english-tweet-tokenizer
jcollado/english-tweet-tokenizer
english-tweet-tokenizer is a machine learning model from jcollado. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This tokenizer has been trained with tweets that have been preprocessed as follows:
Access
Public
Updated Mar 23, 2022
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.3 MB · 86%
At a glance
- Access
- Public
- Created
- Mar 23, 2022
- Updated
- Mar 23, 2022
- SHA
- a6beb474
Model card
From the Hugging Face model README
Text preprocessing
This tokenizer has been trained with tweets that have been preprocessed as follows:
- User mentions (@user_name) have been replaced with the word user.
- URLs have been replace with the word url.
- WIP.
If you are going to use this tokenizer, we recommend you to preprocess your own dataset in the same manner.
- Created
- Mar 23, 2022
- Updated
- Mar 23, 2022