Downloads · 30 days
83
2% of all-time downloads
RichardErkhov/beomi_-_OPEN-SOLAR-KO-10.7B-gguf
beomi_-_OPEN-SOLAR-KO-10.7B-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
83
2% of all-time downloads
All-time downloads
3.5K
Public
Repo size
112 GB
Likes
0
Public
Click a slice to open those files.
.gguf131 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
OPEN-SOLAR-KO-10.7B - GGUF
| Name | Quant method | Size |
|---|---|---|
| OPEN-SOLAR-KO-10.7B.Q2_K.gguf | Q2_K | 3.79GB |
| OPEN-SOLAR-KO-10.7B.IQ3_XS.gguf | IQ3_XS | 4.21GB |
| OPEN-SOLAR-KO-10.7B.IQ3_S.gguf | IQ3_S | 4.44GB |
| OPEN-SOLAR-KO-10.7B.Q3_K_S.gguf | Q3_K_S | 4.41GB |
| OPEN-SOLAR-KO-10.7B.IQ3_M.gguf | IQ3_M | 4.58GB |
| OPEN-SOLAR-KO-10.7B.Q3_K.gguf | Q3_K | 4.91GB |
| OPEN-SOLAR-KO-10.7B.Q3_K_M.gguf | Q3_K_M | 4.91GB |
| OPEN-SOLAR-KO-10.7B.Q3_K_L.gguf | Q3_K_L | 5.33GB |
| OPEN-SOLAR-KO-10.7B.IQ4_XS.gguf | IQ4_XS | 5.5GB |
| OPEN-SOLAR-KO-10.7B.Q4_0.gguf | Q4_0 | 5.73GB |
| OPEN-SOLAR-KO-10.7B.IQ4_NL.gguf | IQ4_NL | 5.8GB |
| OPEN-SOLAR-KO-10.7B.Q4_K_S.gguf | Q4_K_S | 5.78GB |
| OPEN-SOLAR-KO-10.7B.Q4_K.gguf | Q4_K | 6.1GB |
| OPEN-SOLAR-KO-10.7B.Q4_K_M.gguf | Q4_K_M | 6.1GB |
| OPEN-SOLAR-KO-10.7B.Q4_1.gguf | Q4_1 | 6.35GB |
| OPEN-SOLAR-KO-10.7B.Q5_0.gguf | Q5_0 | 6.97GB |
| OPEN-SOLAR-KO-10.7B.Q5_K_S.gguf | Q5_K_S | 6.97GB |
| OPEN-SOLAR-KO-10.7B.Q5_K.gguf | Q5_K | 7.16GB |
| OPEN-SOLAR-KO-10.7B.Q5_K_M.gguf | Q5_K_M | 7.16GB |
| OPEN-SOLAR-KO-10.7B.Q5_1.gguf | Q5_1 | 7.59GB |
| OPEN-SOLAR-KO-10.7B.Q6_K.gguf | Q6_K | 8.29GB |
language:
Update Log
Solar-Ko represents an advanced iteration of the upstage/SOLAR-10.7B-v1.0 model, featuring an expanded vocabulary and the inclusion of a Korean corpus for enhanced pretraining.
Open-Solar-Ko exclusively utilizes publicly accessible Korean corpora, including sources such as AI Hub, Modu Corpus, 모두의 말뭉치, and Korean Wikipedia.
As training was conducted solely with publicly available corpora, this model is open for unrestricted use by everyone, adhering to the Apache2.0 open source License.
Model Developers: Junbum Lee (Beomi)
Variations: Solar-Ko is available with one parameter sizes — 10B with Continual Pretrained version.
Input: The model accepts only text input.
Output: The model produces text output exclusively.
Model Architecture:
SOLAR-KO-10.7B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.
| Training Data | Parameters | Content Length | GQA | Tokens | Learning Rate | |
|---|---|---|---|---|---|---|
| SOLAR-KO-10.7B | A curated mix of Publicly Accessible Korean Corpora | 10.7B | 4k | O | >15B* | 5e<sup>-5</sup> |
Training Corpus
The model was trained using selected datasets from AIHub and Modu Corpus. Detailed information about the training datasets is available below:
Training segment of the data was used.Validation and Test segments were deliberately excluded.The final JSONL dataset used to train this model is approximately 61GB in size.
Total token count: Approximately 15 billion tokens (*using the expanded tokenizer. With the original SOLAR tokenizer, >60 billion tokens.)
Vocab Expansion
| Model Name | Vocabulary Size | Description |
|---|---|---|
| Original Solar | 32000 | Sentencepiece BPE |
| Expanded SOLAR-KO-10.7B | 46592 | Sentencepiece BPE. Added Korean vocab and merges |
Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."
| Model | Tokens |
|---|---|
| SOLAR-10.7B | ['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '날', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '좋', '네', '요', '.'] |
| SOLAR-KO-10.7B | ['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요', '.'] |
Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"
| Model | Tokens |
|---|---|
| SOLAR-10.7B | ['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!'] |
| SOLAR-KO-10.7B | ['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!'] |
Apache 2.0
| 0 | 5 | 10 | 50 | |
|---|---|---|---|---|
| kobest_boolq (macro_f1) | 0.853949 | 0.88098 | 0.898139 | 0.902354 |
| kobest_copa (macro_f1) | 0.804531 | 0.826736 | 0.837656 | 0.860899 |
| kobest_hellaswag (macro_f1) | 0.507174 | 0.500983 | 0.487287 | 0.512182 |
| kobest_sentineg (macro_f1) | 0.3517 | 0.972291 | 0.977321 | 0.984884 |
| kohatespeech (macro_f1) | 0.258111 | 0.403957 | 0.386808 | 0.462393 |
| kohatespeech_apeach (macro_f1) | 0.337667 | 0.651697 | 0.705337 | 0.827757 |
| kohatespeech_gen_bias (macro_f1) | 0.124535 | 0.503464 | 0.498501 | 0.443218 |
| korunsmile (f1) | 0.3814 | 0.356939 | 0.369989 | 0.296193 |
| nsmc (acc) | 0.5356 | 0.87162 | 0.88654 | 0.89632 |
| pawsx_ko (acc) | 0.5435 | 0.5245 | 0.5315 | 0.5385 |
@misc {solar_ko_junbum_2023,
author = { {L. Junbum} },
title = { Solar-Ko-10.7b },
year = 2024,
url = { https://huggingface.co/beomi/SOLAR-KO-10.7B },
publisher = { Hugging Face }
}