Downloads · 30 days
0
distilgpt2
GuillaumeSalouHF/distilgpt2
distilgpt2 is a machine learning model from GuillaumeSalouHF. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
DistilGPT2 English language model pretrained with the supervision of GPT2 (the smallest version of GPT2) on OpenWebTextCorpus, a reproduction of OpenAI's WebText dataset. The model has 6 layers, 768 dimension and 12 h…
Access
Public
Updated May 23, 2022
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.4 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- May 23, 2022
- Updated
- May 23, 2022
- SHA
- dc1cb92d
Datasets
Model card
From the Hugging Face model README
DistilGPT2
test
DistilGPT2 English language model pretrained with the supervision of GPT2 (the smallest version of GPT2) on OpenWebTextCorpus, a reproduction of OpenAI's WebText dataset. The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters (compared to 124M parameters for GPT2). On average, DistilGPT2 is two times faster than GPT2.
On the WikiText-103 benchmark, GPT2 reaches a perplexity on the test set of 16.3 compared to 21.1 for DistilGPT2 (after fine-tuning on the train set).
We encourage to check GPT2 to know more about usage, limitations and potential biases.
<a href="https://huggingface.co/exbert/?model=distilgpt2"> <img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png"> </a>- License
- apache-2.0
- Languages
- en
- Created
- May 23, 2022
- Updated
- May 23, 2022