Downloads · 30 days
25
1% of all-time downloads
gpt-sl-base
cjvt/gpt-sl-base
gpt-sl-base is a text generation model from cjvt. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a Slovene GPT model, based on the bigscience workshop fork of the Megatron. GPT-sl-base was trained on large Slovene corpora: Gigafida, KAS, slWaC, and MaCoCu.
All-time downloads
1.7K
Public
Repo size
842 MB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin289 MB · 52%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- gpt2
- Access
- Public
- Created
- Nov 8, 2022
- Updated
- Nov 8, 2022
- SHA
- 9fbec628
Model card
From the Hugging Face model README
GPT-sl-base
This model is a Slovene GPT model, based on the bigscience workshop fork of the Megatron. GPT-sl-base was trained on large Slovene corpora: Gigafida, KAS, slWaC, and MaCoCu.
Model architecture
GPT-sl-base has about 110 million parameters. It consists of 12 transformer layers with a dimension of 768. It has 16 attention heads and can process sequences up to 1024 tokens in length. The tokenizer was trained on a smaller subset of the corpora, and has the vocabulary of 60k tokens.
Training
The model was trained for about 20 epochs, a total of 390k steps or 102B tokens seen during training.
| Step | Validation Perplexity |
|---|---|
| 50000 | 26.801 |
| 100000 | 25.574 |
| 150000 | 24.773 |
| 200000 | 24.099 |
| 250000 | 23.336 |
| 300000 | 22.607 |
| 350000 | 22.329 |
| 390000 | 22.293 |
- Task
- Text Generation
- Library
- transformers
- Type
- gpt2
- License
- apache-2.0
- Languages
- sl
- Created
- Nov 8, 2022
- Updated
- Nov 8, 2022