Yi-Ko-DUS-9B

beomi/Yi-Ko-DUS-9B

Yi-Ko-DUS-9B is a text generation model from beomi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.

Update @ 2024.01.29 Released Yi-Ko(KoEN)-DUS-9B model 🎉

Downloads · 30 days

199

10% of all-time downloads

All-time downloads

Public

Parameters

8.9B

17.9 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors17.9 GB · 100%

At a glance

Task: Text Generation
Library: transformers
License: apache-2.0
Model type: llama
Access: Public
Created: Jan 29, 2024
Updated: Jul 8, 2024
SHA: 8777cfb8

Try a prompt

Model card

From the Hugging Face model README

Update @ 2024.01.29 Released Yi-Ko(KoEN)-DUS-9B model 🎉

beomi/Yi-Ko-DUS-9B

Yi-Ko-DUS model serves as DUS-applied and advanced iterations of beomi/Yi-Ko-6B model, benefiting from an expanded vocabulary and the inclusion of Korean/English corpus in its further pretraining.

Yi-Ko-DUS model operates with 9B billion parameters.

This repository focuses on the 9B pretrained version, which is tailored to fit the Hugging Face Transformers format, trained after DUS method applied.

Model Details

Model Developers Junbum Lee (Beomi), Taekyoon Choi (Taekyoon)

Variations Yi-Ko-DUS has 9B model only.

Input Models input text only.

Output Models generate text only.

Model Architecture

Yi-Ko-DUS series models are an auto-regressive language model that uses an optimized transformer architecture based on Llama-2*.

<small>*Yi model architecture is based on Llama2, so it can be loaded via LlamaForCausalLM class on HF.</small>

Model Name	Training Data	Params	Context Length	GQA	Trained Tokens	LR	Batch Size(per step)
Yi-Ko-DUS-9B	A mix of Korean + English online data	9B	4k	O	>120B	5e<sup>-5</sup>	2M tokens

Vocab Expansion

Model Name	Vocabulary Size	Description
Original Yi-Series	64000	Sentencepiece BPE
Expanded Yi-Ko(DUS) Series	78464	Sentencepiece BPE. Added Korean vocab and merges

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요.ㅎㅎ"

Model	# of tokens	Tokens
Original Yi-Series	47	`['<0xEC>', '<0x95>', '<0x88>', '<0xEB>', '<0x85>', '<0x95>', '하', '<0xEC>', '<0x84>', '<0xB8>', '<0xEC>', '<0x9A>', '<0x94>', ',', '▁', '<0xEC>', '<0x98>', '<0xA4>', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '<0xEB>', '<0x82>', '<0xA0>', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '<0xEC>', '<0xA2>', '<0x8B>', '<0xEB>', '<0x84>', '<0xA4>', '<0xEC>', '<0x9A>', '<0x94>', '.', '<0xE3>', '<0x85>', '<0x8E>', '<0xE3>', '<0x85>', '<0x8E>']`
Expanded Yi-Ko(DUS) Series	10	`['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요', '.', 'ㅎ', 'ㅎ']`
<small>*Equal Korean vocab with Llama-2-Ko Series</small>

Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"

Model	# of tokens	Tokens
Original Yi-Series	21	`['The', '▁Y', 'i', '▁series', '▁models', '▁are', '▁large', '▁language', '▁models', '▁trained', '▁from', '▁scratch', '▁by', '▁developers', '▁at', '▁', '0', '1', '.', 'AI', '.']`
Expanded Yi-Ko(DUS) Series	21	`['▁The', '▁Y', 'i', '▁series', '▁models', '▁are', '▁large', '▁language', '▁models', '▁trained', '▁from', '▁scratch', '▁by', '▁developers', '▁at', '▁', '0', '1', '.', 'AI', '.']`
<small>*Equal Korean vocab with Llama-2-Ko Series</small>		<small>Since Expanded Yi-Ko Series* prepends `_` at the beginning of the text(to ensure same tokenization for Korean sentences), it shows negilible difference for the first token on English tokenization. </small>

Model Benchmark

5-shot Korean Dataset Evaluation

KMMLU: 43.3514 (exact_match, kmmlu_direct)

+2.58%p than beomi/Yi-Ko-6B

KorQuAD: 80.8798 (exact_match)

+3.06%p than beomi/Yi-Ko-6B

NSMC: 88.352 (acc)

+0.3%p than beomi/Yi-Ko-6B

KOBEST COPA: 84.4831 (macro_f1)

+3.6%p than beomi/Yi-Ko-6B

KOBEST HellaSwag: 52.6099 (macro_f1)

+2.7%p than beomi/Yi-Ko-6B

Apeach: Korean HateSpeech: 63.4723 (macro_f1)

+13.6%p than beomi/Yi-Ko-6B

LICENSE

Apache 2.0 (for research)

For commercial purpose, mailto: [email protected] to acquire Yi-Ko sereis commercial license.

Citation

Please use this bibtex below:

@misc {lee_junbum_2024,
	author       = { {Lee Junbum, Choi Taekyoon} },
	title        = { Yi-Ko-DUS-9B },
	year         = 2024,
	url          = { https://huggingface.co/beomi/Yi-Ko-DUS-9B },
	doi          = { 10.57967/hf/1707 },
	publisher    = { Hugging Face }
}

Acknowledgement

The training is supported by TPU Research Cloud program.

Task: Text Generation
Library: transformers
Type: llama
License: apache-2.0
Languages: en, ko
Created: Jan 29, 2024
Updated: Jul 8, 2024