PathFinderKR_-_Waktaverse-Llama-3-KO-8B-Instruct-4bits

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Waktaverse-Llama-3-KO-8B-Instruct - bnb 4bits

• Model creator: https://huggingface.co/PathFinderKR/
• Original model: https://huggingface.co/PathFinderKR/Waktaverse-Llama-3-KO-8B-Instruct/

Original model description:

language:

• ko
• en license: llama3 library_name: transformers datasets:
• MarkrAI/KoCommercial-Dataset

---

Waktaverse-Llama-3-KO-8B-Instruct Model Card

Model Details

Waktaverse-Llama-3-KO-8B-Instruct is a state-of-the-art Korean language model developed by Waktaverse AI team. This large language model is a specialized version of the Meta-Llama-3-8B-Instruct, tailored for Korean natural language processing tasks. It is designed to handle a variety of complex instructions and generate coherent, contextually appropriate responses.

• Developed by: Waktaverse AI
• Model type: Large Language Model
• Language(s) (NLP): Korean, English
• License: Llama3
• Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct

Model Sources

• Repository: GitHub
• Paper : [More Information Needed]

Uses

Direct Use

The model can be utilized directly for tasks such as text completion, summarization, and question answering without any fine-tuning.

Out-of-Scope Use

This model is not intended for use in scenarios that involve high-stakes decision-making including medical, legal, or safety-critical areas due to the potential risks of relying on automated decision-making. Moreover, any attempt to deploy the model in a manner that infringes upon privacy rights or facilitates biased decision-making is strongly discouraged.

Bias, Risks, and Limitations

While Waktaverse Llama 3 is a robust model, it shares common limitations associated with machine learning models including potential biases in training data, vulnerability to adversarial attacks, and unpredictable behavior under edge cases. There is also a risk of cultural and contextual misunderstanding, particularly when the model is applied to languages and contexts it was not specifically trained on.

How to Get Started with the Model

You can run conversational inference using the Transformers Auto classes. We highly recommend that you add Korean system prompt for better output. Adjust the hyperparameters as you need.

Example Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = (
    "cuda:0" if torch.cuda.is_available() else # Nvidia GPU
    "mps" if torch.backends.mps.is_available() else # Apple Silicon GPU
    "cpu"
)

model_id = "PathFinderKR/Waktaverse-Llama-3-KO-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device=device,
)

################################################################################
# Generation parameters
################################################################################
num_return_sequences=1
max_new_tokens=1024
temperature=0.9
top_k=40
top_p=0.9
repetition_penalty=1.1

def generate_response(system ,user):
    messages = [
        {"role": "system", "content": system},
        {"role": "user", "content": user}
    ]
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False
    )
    
    input_ids = tokenizer.encode(
        prompt,
        add_special_tokens=False,
        return_tensors="pt"
    ).to(device)
    
    outputs = model.generate(
        input_ids=input_ids,
        pad_token_id=tokenizer.eos_token_id,
        num_return_sequences=num_return_sequences,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=temperature,
        top_k=top_k,
        top_p=top_p,
        repetition_penalty=repetition_penalty
    )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=False)

system_prompt = "다음 지시사항에 대한 응답을 작성해주세요."
user_prompt = "피보나치 수열에 대해 설명해주세요."
response = generate_response(system_prompt, user_prompt)
print(response)

Example Output

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

다음 지시사항에 대한 응답을 작성해주세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

피보나치 수열에 대해 설명해주세요.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

피보나치 수열은 수학에서 가장 유명한 수열 중 하나로, 0과 1로 시작하는 숫자들의 모임입니다. 각 숫자는 이전 두 개의 숫자의 합으로 정의되며, 이렇게 계속 반복됩니다. 피보나치 수열은 무한히 커지는데, 첫 번째와 두 번째 항이 모두 0일 수도 있지만 일반적으로는 첫 번째 항이 1이고 두 번째 항이 1입니다.

예를 들어, 0 + 1 = 1, 1 + 1 = 2, 2 + 1 = 3, 3 + 2 = 5, 5 + 3 = 8, 8 + 5 = 13, 13 + 8 = 21, 21 + 13 = 34 등이 있습니다. 이 숫자들을 피보나치 수열이라고 합니다.

피보나치 수열은 다른 수열들과 함께 사용될 때 도움이 됩니다. 예를 들어, 금융 시장에서는 금리 수익률을 나타내기 위해 이 수열이 사용됩니다. 또한 컴퓨터 과학과 컴퓨터 과학에서도 종종 찾을 수 있습니다. 피보나치 수열은 매우 복잡하며 많은 숫자가 나오므로 일반적인 수열처럼 쉽게 구할 수 없습니다. 이 때문에 피보나치 수열은 대수적 함수와 관련이 있으며 수학자들은 이를 연구하고 계산하기 위해 다양한 알고리즘을 개발했습니다.

참고 자료: https://en.wikipedia.org/wiki/Fibonacci_sequence#Properties.<|eot_id|>

Training Details

Training Data

The model is trained on the MarkrAI/KoCommercial-Dataset, which consists of various commercial texts in Korean.

Training Procedure

The model training used LoRA for computational efficiency. 0.02 billion parameters(0.26% of total parameters) were trained.

Training Hyperparameters

################################################################################
# bitsandbytes parameters
################################################################################
load_in_4bit=True
bnb_4bit_compute_dtype=torch_dtype
bnb_4bit_quant_type="nf4"
bnb_4bit_use_double_quant=False

################################################################################
# LoRA parameters
################################################################################
task_type="CAUSAL_LM"
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
r=8
lora_alpha=16
lora_dropout=0.05
bias="none"

################################################################################
# TrainingArguments parameters
################################################################################
num_train_epochs=1
per_device_train_batch_size=1
per_device_eval_batch_size=2
gradient_accumulation_steps=4
gradient_checkpointing=True
learning_rate=2e-5
lr_scheduler_type="cosine"
warmup_ratio=0.1
weight_decay=0.1

################################################################################
# SFT parameters
################################################################################
max_seq_length=1024
packing=True

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

Results

[More Information Needed]

Summary

Technical Specifications

Compute Infrastructure

Hardware

• GPU: NVIDIA GeForce RTX 4080 SUPER

Software

• Operating System: Linux
• Deep Learning Framework: Hugging Face Transformers, PyTorch

Training Details

• Training time: 32 hours
• VRAM usage: 12.8 GB
• GPU power usage: 300 W

Citation

Waktaverse-Llama-3

TBD

Llama-3

@article{llama3modelcard,
  title={Llama 3 Model Card},
  author={AI@Meta},
  year={2024},
  url = {https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}
}

Model Card Authors

[More Information Needed]

Model Card Contact

[More Information Needed]