Downloads · 30 days
8
13% of all-time downloads
minipile_1B
washimneupane/minipile_1B
minipile_1B is a machine learning model from washimneupane. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
A decoder only model trained on minipile dataset from huggingface trained using gpt2 like model architecture.The following configuration files are used for training. The training was done on 3 NVIDA A100 GPUs.
All-time downloads
62
Public
Repo size
21.9 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt10.9 GB · 50%
At a glance
- Library
- transformers
- License
- mit
- Model type
- gpt2
- Access
- Public
- Created
- Sep 6, 2023
- Updated
- Oct 13, 2023
- SHA
- a798168c
Model card
From the Hugging Face model README
A decoder only model trained on minipile dataset from huggingface trained using gpt2 like model architecture.The following configuration files are used for training. The training was done on 3 NVIDA A100 GPUs.
batch_size: 4
block_size: 1024
gradient_accumulation_steps: 21
max_iters: 200000
lr_decay_iters: 180000
warmup_iters: 20000 # 10% of max_iters
weight_decay: 0.1
dropout: 0.1
device: 'cuda'
n_layer: 16
n_head: 16
n_embd: 2048
The files pytorch_model.bin contains the final checkpoint for the last iteration. The file "checkpoint_iter60k" contains the intermediate checkpoint at 60k-th iteration.
- Library
- transformers
- Type
- gpt2
- License
- mit
- Created
- Sep 6, 2023
- Updated
- Oct 13, 2023