beomi_-_Yi-Ko-6B-4bits

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Yi-Ko-6B - bnb 4bits

• Model creator: https://huggingface.co/beomi/
• Original model: https://huggingface.co/beomi/Yi-Ko-6B/

Original model description:

language:

• en
• ko library_name: transformers tags:
• pytorch
• Yi-Ko
• 01-ai
• Yi extra_gated_heading: Access beomi/Yi-Ko-6B on Hugging Face extra_gated_button_content: Submit extra_gated_fields: I agree to share my name, email address and username: checkbox ? I confirm that I understand this project is for research purposes only, and confirm that I agree to follow the LICENSE of this model : checkbox pipeline_tag: text-generation inference: false model-index:
• name: Yi-Ko-6B results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 48.89 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=beomi/Yi-Ko-6B name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 74.48 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=beomi/Yi-Ko-6B name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 55.72 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=beomi/Yi-Ko-6B name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 37.09 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=beomi/Yi-Ko-6B name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 72.93 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=beomi/Yi-Ko-6B name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 12.51 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=beomi/Yi-Ko-6B name: Open LLM Leaderboard license: apache-2.0

---

Update @ 2024.01.29 New Model beomi/Yi-Ko-DUS-9B Released! 🎉

Update @ 2023.12.03 Yi-Ko(KoEN)-6B Achieved #1🥇 Pretrained Models at Open Korean LLM Leaderboard! 🎉

Update @ 2023.12.01 Alpha Release of Yi-Ko(KoEN)-6B model 🎉

beomi/Yi-Ko-6B

Yi-Ko series models serve as advanced iterations of 01-ai/Yi models, benefiting from an expanded vocabulary and the inclusion of Korean/English corpus in its further pretraining. Just like its predecessor, Yi-Ko series models operate within the broad range of generative text models that stretch from 6 billion to 34 billion parameters. This repository focuses on the 6B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.

Model Details

Model Developers Junbum Lee (Beomi)

Variations Yi-Ko series will come in a range of parameter sizes — 6B and 34B variations.

Input Models input text only.

Output Models generate text only.

Model Architecture

Yi-Ko series models are an auto-regressive language model that uses an optimized transformer architecture based on Llama-2*.

<small>*Yi model architecture is based on Llama2, so it can be loaded via LlamaForCausalLM class on HF.</small>

Vocab Expansion

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요.ㅎㅎ"

Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

LICENSE

Apache 2.0 (for research)

For commercial purpose, mailto: [email protected] to acquire Yi-Ko sereis commercial license.

Citation

Please use this bibtex below:

@misc {lee_junbum_2024,
	author       = { {Lee Junbum} },
	title        = { Yi-Ko-6B (Revision 205083a) },
	year         = 2024,
	url          = { https://huggingface.co/beomi/Yi-Ko-6B },
	doi          = { 10.57967/hf/1708 },
	publisher    = { Hugging Face }
}

Acknowledgement

The training is supported by TPU Research Cloud program.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here