Downloads · 30 days
25
24% of all-time downloads
dropout_gpt2
danj0nes/dropout_gpt2
dropout_gpt2 is a text generation model from danj0nes. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
This model is a fine-tuned version of gpt2 on a English Wikipedia dataset.
All-time downloads
103
Public
Parameters
124M
3.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors498 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- gpt2
- Access
- Public
- Created
- Jan 6, 2024
- Updated
- Jan 8, 2024
- SHA
- 6c9c3456
Try a prompt
Base models
Model card
From the Hugging Face model README
dropout_gpt2
This model is a fine-tuned version of gpt2 on a English Wikipedia dataset.
Model description
Dropout debiased gpt2 using the hyperparameters specified in Measuring and Reducing Gendered Correlations in Pre-trained Models (Webster et al. 2021) and the code used in An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (Meade et al. 2022).
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 8
- seed: 0
- gradient_accumulation_steps: 32
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 4000
Framework versions
- Transformers 4.35.2
- Pytorch 2.1.0+cu121
- Datasets 2.16.1
- Tokenizers 0.15.0
- Task
- Text Generation
- Library
- transformers
- Type
- gpt2
- License
- mit
- Languages
- en
- Created
- Jan 6, 2024
- Updated
- Jan 8, 2024