Downloads · 30 days
14
5% of all-time downloads
GPT-6B_Tuned_small_pile
Defalt-404/GPT-6B_Tuned_small_pile
GPT-6B_Tuned_small_pile is a text generation model from Defalt-404. Use it when you need the model to write or continue text. It is set up for transformers.
GPT-6BTunedsmallpile is a GPT-j-6B model trained on 0.1 million example of pile dataset.
All-time downloads
270
Public
Repo size
12.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin12.2 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gptj
- Access
- Public
- Created
- May 8, 2023
- Updated
- May 8, 2023
- SHA
- 38a656ec
Try a prompt
Datasets
Model card
From the Hugging Face model README
Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->GPT-6B_Tuned_small_pile is a GPT-j-6B model trained on 0.1 million example of pile dataset.
n_embd: 4096, n_layer: 28, n_positions: 2048
Tuning Parameters:
val_split_percent: 20,
momentum: 0.9
train_batch_size (eff) : 32
train_micro_batch: 16
gradient_accumulation_steps: 2
gradient_clipping: 0.5
learning_rate: 0.00001
weight_decay: 0.01
lr_schedular: cosine
lr_warmup_steps: 1000
lr_decay: 0.1
lr_decay_step: 2000
mixed_precision: bf16
## Model Details
Model Description
<!-- Provide a longer summary of what this model is. -->- Developed by: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: [More Information Needed]
- Language(s) (NLP): [More Information Needed]
- License: [More Information Needed]
- Finetuned from model: EleutherAI/gpt-j-6b
- Task
- Text Generation
- Library
- transformers
- Type
- gptj
- Languages
- en
- Created
- May 8, 2023
- Updated
- May 8, 2023