Downloads · 30 days
0
Jam-CGPT
apcl/Jam-CGPT
Jam-CGPT is a machine learning model from apcl. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Jam-CGPT is a GPT2-like model that follows jam's pretraining procedure to pretrain models ranging from 38 million to 350 million parameters and finetuning with comments generated by GPT-3.5 and data size ranging from…
Access
Public
Updated Aug 12, 2024
Repo size
34.4 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pt34.4 GB · 100%
At a glance
- Access
- Public
- Created
- Aug 11, 2023
- Updated
- Aug 12, 2024
- SHA
- aa1e2fb4
Model card
From the Hugging Face model README
Jam-CGPT
Jam-CGPT is a GPT2-like model that follows jam's pretraining procedure to pretrain models ranging from 38 million to 350 million parameters and finetuning with comments generated by GPT-3.5 and data size ranging from 170k to 2.15m.
Jam-CGPT Training Details
- We follow jam's pretraining procedure and use the same data to pretrain our 38m, 110m and 350m parameters models.
- We finetune our Jam-CGPT with the summaries generated by GPT-3.5 and 4 different dataset size Jam-CGPT dataset.
- We finetune our models for 3 epochs.
- Our GitHub repo contains the code for reproduction using the same data.
Jam-CGPT 38 million parameters model
| Hyperparameter | Description | Value |
|---|---|---|
| e | embedding dimensions | 512 |
| L | number of layers | 4 |
| h | attention heads | 4 |
| c | block size / context length | 256 |
| b | batch size | 64 |
| a | accumulation steps | 2 |
| d | dropout | 0.20 |
| r | learning rate | 3e-5 |
| y | iterations | 1e-5 |
| iter | number of iterations after pretraing | 757,000 |
Jam-CGPT 110 million parameters model
| Hyperparameter | Description | Value |
|---|---|---|
| e | embedding dimensions | 768 |
| L | number of layers | 10 |
| h | attention heads | 8 |
| c | block size / context length | 256 |
| b | batch size | 32 |
| a | accumulation steps | 4 |
| d | dropout | 0.20 |
| r | learning rate | 3e-5 |
| y | iterations | 1e-5 |
| iter | number of iterations after pretraing | 762,000 |
Jam-CGPT 350 million parameters model
| Hyperparameter | Description | Value |
|---|---|---|
| e | embedding dimensions | 1024 |
| L | number of layers | 24 |
| h | attention heads | 16 |
| c | block size / context length | 256 |
| b | batch size | 4 |
| a | accumulation steps | 32 |
| d | dropout | 0.20 |
| r | learning rate | 3e-5 |
| y | weight decay | 1e-5 |
| iter | iterations | 272,000 |
- Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128.
- If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps.
- We pretrained 38m and 110m models for 3 epochs.
- Created
- Aug 11, 2023
- Updated
- Aug 12, 2024