beomi_-_Solar-Ko-Recovery-11B-gguf

RichardErkhov/beomi_-_Solar-Ko-Recovery-11B-gguf

beomi_-_Solar-Ko-Recovery-11B-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

Downloads · 30 days

332

17% of all-time downloads

All-time downloads

1.9K

Public

Repo size

125 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.gguf145 GB · 100%

At a glance

Access: Public
Created: May 25, 2024
Updated: May 25, 2024
SHA: a82cd11e

Model card

From the Hugging Face model README

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Solar-Ko-Recovery-11B - GGUF

Model creator: https://huggingface.co/beomi/
Original model: https://huggingface.co/beomi/Solar-Ko-Recovery-11B/

Name	Quant method	Size
Solar-Ko-Recovery-11B.Q2_K.gguf	Q2_K	3.87GB
Solar-Ko-Recovery-11B.IQ3_XS.gguf	IQ3_XS	4.29GB
Solar-Ko-Recovery-11B.IQ3_S.gguf	IQ3_S	4.52GB
Solar-Ko-Recovery-11B.Q3_K_S.gguf	Q3_K_S	4.5GB
Solar-Ko-Recovery-11B.IQ3_M.gguf	IQ3_M	4.67GB
Solar-Ko-Recovery-11B.Q3_K.gguf	Q3_K	4.99GB
Solar-Ko-Recovery-11B.Q3_K_M.gguf	Q3_K_M	4.99GB
Solar-Ko-Recovery-11B.Q3_K_L.gguf	Q3_K_L	5.42GB
Solar-Ko-Recovery-11B.IQ4_XS.gguf	IQ4_XS	5.59GB
Solar-Ko-Recovery-11B.Q4_0.gguf	Q4_0	5.82GB
Solar-Ko-Recovery-11B.IQ4_NL.gguf	IQ4_NL	5.89GB
Solar-Ko-Recovery-11B.Q4_K_S.gguf	Q4_K_S	5.87GB
Solar-Ko-Recovery-11B.Q4_K.gguf	Q4_K	6.19GB
Solar-Ko-Recovery-11B.Q4_K_M.gguf	Q4_K_M	6.19GB
Solar-Ko-Recovery-11B.Q4_1.gguf	Q4_1	6.45GB
Solar-Ko-Recovery-11B.Q5_0.gguf	Q5_0	7.07GB
Solar-Ko-Recovery-11B.Q5_K_S.gguf	Q5_K_S	7.07GB
Solar-Ko-Recovery-11B.Q5_K.gguf	Q5_K	7.26GB
Solar-Ko-Recovery-11B.Q5_K_M.gguf	Q5_K_M	7.26GB
Solar-Ko-Recovery-11B.Q5_1.gguf	Q5_1	7.7GB
Solar-Ko-Recovery-11B.Q6_K.gguf	Q6_K	8.4GB
Solar-Ko-Recovery-11B.Q8_0.gguf	Q8_0	10.88GB

Original model description:

language:

ko
en pipeline_tag: text-generation inference: false tags:
solar
mistral
pytorch
solar-ko library_name: transformers license: apache-2.0

Update Log

2024.05.16: Released Solar-Ko-Recovery

Solar-Ko-Recovery-11B 🌟❤️‍🩹

Solar-Ko-Recovery-11B aimed to recover Solar's capability on Korean with re-arrange of Embeddings and LM head, featuring an expanded vocabulary and the inclusion of a Korean+English corpus for enhanced representation.

Model Details

Model Developers: Junbum Lee (Beomi)

Variations: Solar-Ko-Recovery is available with one parameter sizes — 11B(10.99B🤣).

Input: The model accepts only text input.

Output: The model produces text output exclusively.

Model Architecture:

Solar-Ko-Recovery is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.

	Training Data	Parameters	Content Length	GQA	Tokens	Learning Rate
Solar-Ko-Recovery	A curated mix of Korean+English Corpora	10.8B	4k	O	>30B*	5e<sup>-5</sup>

NOTE: Only Embedding layer and LM Head layer are trained.

Vocab Expansion

Vocab expansion is conducted on edited upstage/solar-1-mini-tokenizer, which is superset of Solar tokenizer.

Model Name	Vocabulary Size	Description
Original Solar	32000	Sentencepiece BPE
solar-1-mini-tokenizer	64000	Sentencepiece BPE. Added Ko/JP vocabs

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

SOLAR-10.7B: 26 tokens
Solar-Ko-Recovery: 7 tokens

Model	Tokens
SOLAR-10.7B	`['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '날', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '좋', '네', '요', '.']`
Solar-Ko-Recovery	`['▁안녕하세요', ',', '▁오늘은', '▁날씨가', '▁좋', '네요', '.']`

Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"

SOLAR-10.7B: 22 tokens
Solar-Ko-Recovery: 22 tokens

Model	Tokens
SOLAR-10.7B	`['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']`
Solar-Ko-Recovery	`['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']`

LICENSE

Apache 2.0

Model Benchmark

LM Eval Harness - Korean

Used EleutherAI's lm-evaluation-harness
5-shot scores

TBD

Citation

TBD

Acknowledgements

Training support was provided by the TPU Research Cloud program.

Created: May 25, 2024
Updated: May 25, 2024