Downloads · 30 days
60
12% of all-time downloads
distilbert-tokenizer_256k-MLM_500k
vocab-transformers/distilbert-tokenizer_256k-MLM_500k
distilbert-tokenizer_256k-MLM_500k is a fill-mask model from vocab-transformers. Use it when you need the model to fill a missing word. It is set up for transformers.
This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC,…
All-time downloads
520
Public
Repo size
1.9 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin962 MB · 99%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- distilbert
- Access
- Public
- Created
- Apr 7, 2022
- Updated
- Apr 7, 2022
- SHA
- 39f88754
Try a prompt
Model card
From the Hugging Face model README
DistilBERT with 256k token embeddings
This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC, for 3 epochs.
Then the model was trained on this dataset with MLM for 500k steps (batch size 64). The token embeddings were updated during MLM.
For the same model but with frozen token embeddings while MLM training see: https://huggingface.co/vocab-transformers/distilbert-word2vec_256k-MLM_500k
- Task
- Fill-Mask
- Library
- transformers
- Type
- distilbert
- Created
- Apr 7, 2022
- Updated
- Apr 7, 2022