Downloads · 30 days
66
5% of all-time downloads
pico-gpt-j-6.7m
crumb/pico-gpt-j-6.7m
pico-gpt-j-6.7m is a text generation model from crumb. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as bigscience-bloom-rail-1.0.
All-time downloads
1.3K
Public
Parameters
6.8M
268 MB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.pt53.6 MB · 48%
Parameter types
How the weights are stored.
F326.7M · 99%
Try a prompt
Datasets
Model card
From the Hugging Face model README
6.7m (6,700,128) param GPT-J model.
n_positions - 128
n_embd - 64
n_layer - 4
n_head - 8
rotary_dim - 64
tokenizer - gpt-j
First, trained on 4,194,304 samples from the c4 dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048. A cosine learning rate schedule was used starting at 1e-3.
- Task
- Text Generation
- Library
- transformers
- Type
- gptj
- License
- bigscience-bloom-rail-1.0
- Languages
- en
- Created
- Jan 26, 2023
- Updated
- Mar 17, 2023