Downloads · 30 days
193
0% of all-time downloads
gpt3-finnish-13B
TurkuNLP/gpt3-finnish-13B
gpt3-finnish-13B is a text generation model from TurkuNLP. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Generative Pretrained Transformer with 13B parameteres for Finnish.
All-time downloads
58.2K
Public
Repo size
143 GB
Likes
14
Public
Repo makeup
Click a slice to open those files.
.bin53 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- bloom
- Access
- Public
- Created
- Feb 16, 2023
- Updated
- Jun 27, 2023
- SHA
- ade35fd7
Model card
From the Hugging Face model README
Generative Pretrained Transformer with 13B parameteres for Finnish.
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture. Note that the models are pure language models, meaning that they are not instruction finetuned for dialogue or answering questions.
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
All models are trained for 300B tokens.
Parameters
| Model | Layers | Dim | Heads | Params |
|---|---|---|---|---|
| Small | 12 | 768 | 12 | 186M |
| Medium | 24 | 1024 | 16 | 437M |
| Large | 24 | 1536 | 16 | 881M |
| XL | 24 | 2064 | 24 | 1.5B |
| ”3B” | 32 | 2560 | 32 | 2.8B |
| ”8B” | 32 | 4096 | 32 | 7.5B |
| "13B" | 40 | 5120 | 40 | 13.3B |
Datasets
We used a combination of multiple Finnish resources.
- Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4
- Common Crawl Finnish https://TODO
- Finnish Wikipedia https://fi.wikipedia.org/wiki
- Lönnrot Projekti Lönnrot http://www.lonnrot.net/
- ePub National library ”epub” collection
- National library ”lehdet” collection
- Suomi24 The Suomi 24 Corpus 2001-2020 http://urn.fi/urn:nbn:fi:lb-2021101527
- Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi
- STT Finnish News Agency Archive 1992-2018 http://urn.fi/urn:nbn:fi:lb-2019041501
- Yle Finnish News Archive 2011-2018 http://urn.fi/urn:nbn:fi:lb-2017070501
- Yle Finnish News Archive 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050401
- Yle News Archive Easy-to-read Finnish 2011-2018 http://urn.fi/urn:nbn:fi:lb-2019050901
- Yle News Archive Easy-to-read Finnish 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050701
- ROOTS TODO
Sampling ratios
| Dataset | Chars | Ratio | Weight | W.Ratio |
|---|---|---|---|---|
| Parsebank | 35.0B | 16.9% | 1.5 | 22.7% |
| mC4-Fi | 46.3B | 22.4% | 1.0 | 20.0% |
| CC-Fi | 79.6B | 38.5% | 1.0 | 34.4% |
| Fiwiki | 0.8B | 0.4% | 3.0 | 1.0% |
| Lönnrot | 0.8B | 0.4% | 3.0 | 1.0% |
| Yle | 1.6B | 0.8% | 2.0 | 1.4% |
| STT | 2.2B | 1.1% | 2.0 | 1.9% |
| ePub | 13.5B | 6.5% | 1.0 | 5.8% |
| Lehdet | 5.8B | 2.8% | 1.0 | 2.5% |
| Suomi24 | 20.6B | 9.9% | 1.0 | 8.9% |
| Reddit-Fi | 0.7B | 0.4% | 1.0 | 0.3% |
| TOTAL | 207.0B | 100.0% | N/A | 100.0% |
More documentation and a paper coming soon.
- Task
- Text Generation
- Library
- transformers
- Type
- bloom
- License
- apache-2.0
- Languages
- fi
- Created
- Feb 16, 2023
- Updated
- Jun 27, 2023