beomi_-_llama-2-ko-7b-4bits

Quantization made by Richard Erkhov.

Github

Discord

Request more models

llama-2-ko-7b - bnb 4bits

• Model creator: https://huggingface.co/beomi/
• Original model: https://huggingface.co/beomi/llama-2-ko-7b/

Original model description:

language:

• en
• ko pipeline_tag: text-generation inference: false tags:
• facebook
• meta
• pytorch
• llama
• llama-2
• kollama
• llama-2-ko

---

Update Log

• 2023.12.27
  • New Model is here! Trained with only open-accessible Korean text corpus: https://huggingface.co/beomi/open-llama-2-ko-7b
• 2023.10.19
  • Fix Tokenizer bug(space not applied when decoding) after transforemrs>=4.34.0

Llama-2-Ko 🦙🇰🇷

Llama-2-Ko serves as an advanced iteration of Llama 2, benefiting from an expanded vocabulary and the inclusion of a Korean corpus in its further pretraining. Just like its predecessor, Llama-2-Ko operates within the broad range of generative text models that stretch from 7 billion to 70 billion parameters. This repository focuses on the 7B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.

Model Details

Model Developers Junbum Lee (Beomi)

Variations Llama-2-Ko will come in a range of parameter sizes — 7B, 13B, and 70B — as well as pretrained and fine-tuned variations.

Input Models input text only.

Output Models generate text only.

Model Architecture

Llama-2-Ko is an auto-regressive language model that uses an optimized transformer architecture based on Llama-2.

Vocab Expansion

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

• Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot

NSMC (Acc) - 50000 full test

TBD

COPA (F1)

<img src=https://user-images.githubusercontent.com/11323660/255575809-c037bc6e-0566-436a-a6c1-2329ac92187a.png style="max-width: 700px; width: 100%" />

HellaSwag (F1)

<img src=https://user-images.githubusercontent.com/11323660/255576090-a2bfc1ae-d117-44b7-9f7b-262e41179ec1.png style="max-width: 700px; width: 100%" />

BoolQ (F1)

<img src=https://user-images.githubusercontent.com/11323660/255576343-5d847a6f-3b6a-41a7-af37-0f11940a5ea4.png style="max-width: 700px; width: 100%" />

SentiNeg (F1)

<img src=https://user-images.githubusercontent.com/11323660/255576572-b005a81d-fa4d-4709-b48a-f0fe4eed17a3.png style="max-width: 700px; width: 100%" />

Note for oobabooga/text-generation-webui

Remove ValueError at load_tokenizer function(line 109 or near), in modules/models.py.

diff --git a/modules/models.py b/modules/models.py
index 232d5fa..de5b7a0 100644
--- a/modules/models.py
+++ b/modules/models.py
@@ -106,7 +106,7 @@ def load_tokenizer(model_name, model):
                 trust_remote_code=shared.args.trust_remote_code,
                 use_fast=False
             )
-        except ValueError:
+        except:
             tokenizer = AutoTokenizer.from_pretrained(
                 path_to_model,
                 trust_remote_code=shared.args.trust_remote_code,

Since Llama-2-Ko uses FastTokenizer provided by HF tokenizers NOT sentencepiece package, it is required to use use_fast=True option when initialize tokenizer.

Apple Sillicon does not support BF16 computing, use CPU instead. (BF16 is supported when using NVIDIA GPU)

Citation

@misc {l._junbum_2023,
	author       = { {L. Junbum} },
	title        = { llama-2-ko-7b (Revision 4a9993e) },
	year         = 2023,
	url          = { https://huggingface.co/beomi/llama-2-ko-7b },
	doi          = { 10.57967/hf/1098 },
	publisher    = { Hugging Face }
}

Acknowledgement

The training is supported by TPU Research Cloud program.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here