Downloads · 30 days
0
mega-wikitext-103
mnaylor/mega-wikitext-103
mega-wikitext-103 is a fill-mask model from mnaylor. Use it when you need the model to fill a missing word. The card lists the license as mit.
NOTE: THIS MODEL IS NOT INTEGRATED WITH HUGGING FACE. Please use the version of this model converted to the newly implemented Mega architecture in transformers (link)
Access
Public
Updated Feb 23, 2023
Repo size
55.3 MB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.pt55.3 MB · 96%
At a glance
- Task
- Fill-Mask
- License
- mit
- Access
- Public
- Created
- Dec 21, 2022
- Updated
- Feb 23, 2023
- SHA
- 58183a3d
Try a prompt
Datasets
Model card
From the Hugging Face model README
NOTE: THIS MODEL IS NOT INTEGRATED WITH HUGGING FACE. Please use the version of this model converted to the newly implemented Mega
architecture in transformers (link)
Moving Average Gated Attention (Mega): Pretrained LM
This repo contains pretrained weights for a language model with the Mega architecture (see paper).
I used the Mega source code (namely the MegaEncoderLayer class) and created wrappers for token embeddings and MLM prediction. This model
was pretrained for 5 epochs (11.3k gradient steps) on wikitext-103, which took roughly 5 hours on a single T4 (in Colab's free tier).
See the Colab notebook for further training details. In order to load the pretrained weights for this model, you'll need to use the Mega repo along with the example code at the end of the Colab notebook.
- Task
- Fill-Mask
- License
- mit
- Languages
- en
- Created
- Dec 21, 2022
- Updated
- Feb 23, 2023