Downloads · 30 days
255
0% of all-time downloads
gpt3-finnish-small
TurkuNLP/gpt3-finnish-small
gpt3-finnish-small is a text generation model from TurkuNLP. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Generative Pretrained Transformer with 186M parameteres for Finnish.
All-time downloads
93K
Public
Repo size
2.2 GB
Likes
13
Public
Repo makeup
Click a slice to open those files.
.bin743 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- bloom
- Access
- Public
- Created
- Feb 15, 2023
- Updated
- Jun 27, 2023
- SHA
- 20a19af4
Model card
From the Hugging Face model README
Generative Pretrained Transformer with 186M parameteres for Finnish.
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture. Note that the models are pure language models, meaning that they are not instruction finetuned for dialogue or answering questions.
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
All models are trained for 300B tokens.
Parameters
| Model | Layers | Dim | Heads | Params |
|---|---|---|---|---|
| Small | 12 | 768 | 12 | 186M |
| Medium | 24 | 1024 | 16 | 437M |
| Large | 24 | 1536 | 16 | 881M |
| XL | 24 | 2064 | 24 | 1.5B |
| ”3B” | 32 | 2560 | 32 | 2.8B |
| ”8B” | 32 | 4096 | 32 | 7.5B |
| "13B" | 40 | 5120 | 40 | 13.3B |
Datasets
We used a combination of multiple Finnish resources.
- Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4
- Common Crawl Finnish https://TODO
- Finnish Wikipedia https://fi.wikipedia.org/wiki
- Lönnrot Projekti Lönnrot http://www.lonnrot.net/
- ePub National library ”epub” collection
- National library ”lehdet” collection
- Suomi24 The Suomi 24 Corpus 2001-2020 http://urn.fi/urn:nbn:fi:lb-2021101527
- Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi
- STT Finnish News Agency Archive 1992-2018 http://urn.fi/urn:nbn:fi:lb-2019041501
- Yle Finnish News Archive 2011-2018 http://urn.fi/urn:nbn:fi:lb-2017070501
- Yle Finnish News Archive 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050401
- Yle News Archive Easy-to-read Finnish 2011-2018 http://urn.fi/urn:nbn:fi:lb-2019050901
- Yle News Archive Easy-to-read Finnish 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050701
- ROOTS TODO
Sampling ratios
| Dataset | Chars | Ratio | Weight | W.Ratio |
|---|---|---|---|---|
| Parsebank | 35.0B | 16.9% | 1.5 | 22.7% |
| mC4-Fi | 46.3B | 22.4% | 1.0 | 20.0% |
| CC-Fi | 79.6B | 38.5% | 1.0 | 34.4% |
| Fiwiki | 0.8B | 0.4% | 3.0 | 1.0% |
| Lönnrot | 0.8B | 0.4% | 3.0 | 1.0% |
| Yle | 1.6B | 0.8% | 2.0 | 1.4% |
| STT | 2.2B | 1.1% | 2.0 | 1.9% |
| ePub | 13.5B | 6.5% | 1.0 | 5.8% |
| Lehdet | 5.8B | 2.8% | 1.0 | 2.5% |
| Suomi24 | 20.6B | 9.9% | 1.0 | 8.9% |
| Reddit-Fi | 0.7B | 0.4% | 1.0 | 0.3% |
| TOTAL | 207.0B | 100.0% | N/A | 100.0% |
More documentation and a paper coming soon.
- Task
- Text Generation
- Library
- transformers
- Type
- bloom
- License
- apache-2.0
- Languages
- fi
- Created
- Feb 15, 2023
- Updated
- Jun 27, 2023