Downloads · 30 days
26
0% of all-time downloads
danube-ko-1.8b-base
jjhsnail0822/danube-ko-1.8b-base
danube-ko-1.8b-base is a text generation model from jjhsnail0822. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
danube-ko-1.8b-base is a continual pre-trained Korean language model based on h2oai/h2o-danube2-1.8b-base.
All-time downloads
13K
Public
Parameters
1.9B
3.7 GB on disk
Likes
3
Public
Repo makeup
Click a slice to open those files.
.safetensors3.7 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mistral
- Access
- Public
- Created
- Jun 19, 2024
- Updated
- Oct 29, 2024
- SHA
- f5b62ff7
Base models
Datasets
Model card
From the Hugging Face model README
Model Details
danube-ko-1.8b-base is a continual pre-trained Korean language model based on h2oai/h2o-danube2-1.8b-base.
Model Developers
Jinhong Jeong, Ungsang Yoon
Model Architecture
The vocabulary size was expanded from original 32000 to 40000 to add Korean tokens efficiently. We used the EEVE technique for training. The model has sequence length of 2048. Everything else is the same as the original model.
Training Datasets
We used CulturaX, Common Crawl CC-MAIN-2024-10, AI Hub Data, Korean Wikis, Corpora from National Institute of the Korean Language, Standard Korean Dictionary, etc. About 42GB of data was used for training.
Model Benchmark
This model is ranked #1 in Ko-MMLU on the Open Ko-LLM Leaderboard among pretrained Korean models of size 2B or smaller as of July 5, 2024.
| Task | Value |
|---|---|
| Ko-ARC | 31.74 |
| Ko-HellaSwag | 44.44 |
| Ko-MMLU | 28.06 |
| Ko-TruthfulQA | 41.63 |
| Ko-CommonGen V2 | 32.7 |
| kmmlu_direct | 29.05 |
| kobest | 59.13 |
Disclaimer
The Model can generate information that is biased, discriminatory, socially inappropriate, etc. The Model can also generate information that is not accurate. The Model is used at your own risk, and the developers are not responsible for the information generated by the model.
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- apache-2.0
- Languages
- ko, en
- Created
- Jun 19, 2024
- Updated
- Oct 29, 2024