EEVE-Korean-Instruct-10.8B-v1.0-int8

EEVE-Korean-Instruct-10.8B-v1.0-int8

• Orginal model : yanolja/EEVE-Korean-Instruct-10.8B-v1.0
• Quantized using intel/auto-round

Description

EEVE-Korean-Instruct-10.8B-v1.0-int8 is an int8 model with group_size 128 of the yanolja/EEVE-Korean-Instruct-10.8B-v1.0.

The above model was quantized using AutoRound(Advanced Weight-Only Quantization Algorithm for LLMs) released by intel.

you can find out more in detail through the the GitHub Repository.

Training details

Cloning a repository(AutoRound)

git clone https://github.com/intel/auto-round

Enter into the examples/language-modeling folder

cd auto-round/examples/language-modeling
pip install -r requirements.txt

Here's an simplified code for quantization.

python main.py \
  --model_name "yanolja/EEVE-Korean-Instruct-10.8B-v1.0" \
  --bits 8 \
  --group_size 128 \
  --train_bs 8 \
  --gradient_accumulate_steps 8 \
  --deployment_device 'gpu' \
  --output_dir "./save_ckpt" 

Model inference

Install the necessary packages

pip install auto_gptq
pip install optimum
pip install -U accelerate bitsandbytes datasets peft transformers

Example code

import torch 
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("ssuncheol/EEVE-Korean-Instruct-10.8B-v1.0-int8", device_map = "auto")
tokenizer = AutoTokenizer.from_pretrained("ssuncheol/EEVE-Korean-Instruct-10.8B-v1.0-int8")

device = 'cuda' if torch.cuda.is_available() else 'cpu'

prompt_template = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\nHuman: {prompt}\nAssistant:\n"

text = '한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주'
model_inputs = tokenizer(prompt_template.format(prompt=text), return_tensors='pt').to(device)

outputs = model.generate(**model_inputs, max_new_tokens=256)
output_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
print(output_text)

Example results

Human: 한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.

(A) 경성
(B) 부산
(C) 평양
(D) 서울
(E) 전주

Assistant: 정답은 (D) 서울입니다.