Downloads · 30 days
20
5% of all-time downloads
gpt2_wt103-40m_12-layer
Kristijan/gpt2_wt103-40m_12-layer
gpt2_wt103-40m_12-layer is a machine learning model from Kristijan. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch.
paper: Characterizing Verbatim Short-Term Memory in Neural Language Models
All-time downloads
389
Public
Repo size
1.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt862 MB · 66%
At a glance
- Library
- pytorch
- Model type
- gpt2
- Access
- Public
- Created
- Oct 26, 2022
- Updated
- Oct 26, 2022
- SHA
- 4b86156e
Model card
From the Hugging Face model README
Model description
paper: Characterizing Verbatim Short-Term Memory in Neural Language Models
This is a gpt2-small-like decoder-only transformer model trained on a 40M token subset of the wikitext-103 dataset.
Usage
You can download and load the model as follows:
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained("Kristijan/gpt2_wt103-40m_12-layer")
Alternatively, if you've downloaded the checkpoint files in this repository, you could also do:
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained(path_to_folder_with_checkpoint_files)
To tokenize your text for this model, you should use the tokenizer trained on Wikitext-103
Intended uses
This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.
- Library
- pytorch
- Type
- gpt2
- Languages
- en
- Created
- Oct 26, 2022
- Updated
- Oct 26, 2022