Downloads · 30 days
15
1% of all-time downloads
justinqbui/bertweet-covid19-base-uncased-pretraining-covid-vaccine-tweets
bertweet-covid19-base-uncased-pretraining-covid-vaccine-tweets is a fill-mask model from justinqbui. Use it when you need the model to fill a missing word. It is set up for transformers.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
15
1% of all-time downloads
All-time downloads
1.2K
Public
Repo size
1.6 GB
Likes
0
Public
Click a slice to open those files.
.bin540 MB · 100%
From the Hugging Face model README
This model is a further pre-trained version of vinai/bertweet-covid19-base-uncased on masked language modeling using a kaggle dataset with tweets up until early December. It achieves the following results on the evaluation set (15% from the dataset randomly selected to serve as a test set):
To use the model, use the inference API.
Alternatively, to run locally
from transformers import pipeline
model = "justinqbui/bertweet-covid19-base-uncased-pretraining-covid-vaccine-tweets"
pipe = pipeline("fill-mask", model = model)
seq = "covid vaccines are <mask> and effective"
pipe(seq)
This model is a further pretrained version of bertweet, which both follow objectives in the RoBERTa paper. While bertweet was only trained with 23M tweets until September, 2020, this model was further pre-trained using 300k tweets with #CovidVaccine.
The tokenizer requires the emoji library to be installed.
!pip install nltk emoji
The intended use of this model is for fine-tuning on a downstream task on tasks that are closely related to covid and covid vaccines. This model has many potential biases and limitations, since the model is trained on public tweets, it is bound to recreate biases that people tweet.
In order to load the model and tokenizer, run
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("justinqbui/bertweet-covid19-base-uncased-pretraining-covid-vaccine-tweets")
model = AutoModelForMaskedLM.from_pretrained("justinqbui/bertweet-covid19-base-uncased-pretraining-covid-vaccine-tweets")
This model was further pre-trained on 300k tweets containing #covidvaccines from this kaggle dataset. The evaluation set was 15% of the tweets that were held out from the training data.
See the training notebook found here.
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.5775 | 1.0 | 8931 | 1.5852 |
| 1.5715 | 2.0 | 17862 | 1.5701 |
| 1.5394 | 3.0 | 26793 | 1.5089 |