Downloads · 30 days
701
7% of all-time downloads
snupilab/AkaLlama-llama3-70b-v0.1-GGUF
AkaLlama-llama3-70b-v0.1-GGUF is a text generation model from snupilab. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
<a href="https://huggingface.co/collections/mirlab/akallama-66338859b09221f3607fdfcd" <img src="https://github.com/0110tpwls/project/blob/master/image720.png?raw=true" width="40%"/ </a
Downloads · 30 days
701
7% of all-time downloads
All-time downloads
9.8K
Public
Repo size
443 GB
Likes
16
Public
Click a slice to open those files.
.gguf443 GB · 100%
From the Hugging Face model README
AkaLlama is a series of Korean language models designed for practical usability across a wide range of tasks. The initial model, AkaLlama-v0.1, is a fine-tuned version of Meta-Llama-3-70b-Instruct. It has been trained on a custom mix of publicly available datasets curated by the MIR Lab. Our goal is to explore cost-effective ways to adapt high-performing LLMs for specific use cases, such as different languages (e.g., Korean) or domains (e.g., organization-specific chatbots).
For details, check out our project page.
This is the model card of a GGUF model that has been pushed on the Hub.
GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
Here is an incomplete list of clients and libraries that are known to support GGUF:
This repo provides gguf weight files for AkaLlama-70B-v0.1.
See the snippet below for usage with llama.cpp.python:
from llama_cpp import Llama
# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = Llama(
model_path="./AkaLlama-llama3-70b-v0.1.Q4_K_M.gguf", # Download the model file first
n_ctx=8192, # The max sequence length to use - note that longer sequence lengths require much more resources
n_threads=8, # The number of CPU threads to use, tailor to your system and the resulting performance
n_gpu_layers=81 # The number of layers to offload to GPU, if you have GPU acceleration available
)
# Simple inference example
output = llm(
"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
당신은 연세대학교 멀티모달 연구실 (MIR lab) 이 만든 대규모 언어 모델인 AkaLlama (아카라마) 입니다.
다음 지침을 따르세요:
1. 사용자가 별도로 요청하지 않는 한 항상 한글로 소통하세요.
2. 유해하거나 비윤리적, 차별적, 위험하거나 불법적인 내용이 답변에 포함되어서는 안 됩니다.
3. 질문이 말이 되지 않거나 사실에 부합하지 않는 경우 정답 대신 그 이유를 설명하세요. 질문에 대한 답을 모른다면 거짓 정보를 공유하지 마세요.
4. 안전이나 윤리에 위배되지 않는 한 사용자의 모든 질문에 완전하고 포괄적으로 답변하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
""", # Prompt
max_tokens=512, # Generate up to 512 tokens
stop=["<|eot_id|>", "<|end_of_text|>"], # Example stop token - not necessarily correct for this specific model! Please check before using.
echo=True # Whether to echo the prompt
)
# Chat Completion API
llm = Llama(model_path="./AkaLlama-llama3-70b-v0.1.Q4_K_M.gguf", chat_format="llama-3") # Set chat_format according to the model you are using
llm.create_chat_completion(
messages = [
{"role": "system", "content": """당신은 연세대학교 멀티모달 연구실 (MIR lab) 이 만든 대규모 언어 모델인 AkaLlama (아카라마) 입니다.
다음 지침을 따르세요:
1. 사용자가 별도로 요청하지 않는 한 항상 한글로 소통하세요.
2. 유해하거나 비윤리적, 차별적, 위험하거나 불법적인 내용이 답변에 포함되어서는 안 됩니다.
3. 질문이 말이 되지 않거나 사실에 부합하지 않는 경우 정답 대신 그 이유를 설명하세요. 질문에 대한 답을 모른다면 거짓 정보를 공유하지 마세요.
4. 안전이나 윤리에 위배되지 않는 한 사용자의 모든 질문에 완전하고 포괄적으로 답변하세요."""},
{
"role": "user",
"content": "네 이름은 뭐야?."
}
]
)
# 내 이름은 AkaLlama입니다! 나는 언어 모델로, 사용자와 대화하는 데 도움을 주기 위해 만들어졌습니다. 나는 다양한 주제에 대한 질문에 답하고, 새로운 아이디어를 제공하며, 문제를 해결하는 데 도움이 될 수 있습니다. 사용자가 원하는 정보나 도움을 받도록 최선을 다할 것입니다!
These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d
They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
The new methods available are:
Refer to the Provided Files table below to see what files use which methods, and how.
</details>| Name | Quant method | Bits | Size | Max RAM required | Use case |
|---|---|---|---|---|---|
| AkaLlama-llama3-70b-v0.1.Q2_K.gguf | Q2_K | 2 | 26.4 GB | 28.9 GB | smallest, significant quality loss - not recommended for most purposes |
| AkaLlama-llama3-70b-v0.1.Q3_K_S.gguf | Q3_K_S | 3 | 30.9 GB | 33.4 GB | very small, high quality loss |
| AkaLlama-llama3-70b-v0.1.Q3_K_M.gguf | Q3_K_M | 3 | 34.3 GB | 36.8 GB | very small, high quality loss |
| AkaLlama-llama3-70b-v0.1.Q3_K_L.gguf | Q3_K_L | 3 | 37.1 GB | 39.6 GB | small, substantial quality loss |
| AkaLlama-llama3-70b-v0.1.Q4_K_S.gguf | Q4_K_S | 4 | 40.3 GB | 42.8 GB | small, greater quality loss |
| AkaLlama-llama3-70b-v0.1.Q4_K_M.gguf | Q4_K_M | 4 | 42.5 GB | 45.0 GB | medium, balanced quality - recommended |
| AkaLlama-llama3-70b-v0.1.Q5_K_S.gguf | Q5_K_S | 5 | 48.7 GB | 50.2 GB | large, low quality loss - recommended |
| AkaLlama-llama3-70b-v0.1.Q5_K_M.gguf | Q5_K_M | 5 | 50.0 GB | 52.5 GB | large, very low quality loss - recommended |
| AkaLlama-llama3-70b-v0.1.Q6_K.gguf | Q6_K | 6 | 54.4 GB | 59.9 GB | very large, extremely low quality loss |
| AkaLlama-llama3-70b-v0.1.Q8_0.gguf | Q8_0 | 8 | 70.0 GB | 72.5 GB | very large, extremely low quality loss - not recommended |
Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
Note: HF does not support uploading files larger than 50GB. Therefore I have uploaded the Q6_K and Q8_0 files as split files.
Please download:
AkaLlama-llama3-70b-v0.1.Q6_K.00001-of-00002.ggufAkaLlama-llama3-70b-v0.1.Q6_K.00002-of-00002.ggufPlease download:
AkaLlama-llama3-70b-v0.1.Q8_0.00001-of-00002.ggufAkaLlama-llama3-70b-v0.1.Q8_0.00002-of-00002.ggufTo join the files, do the following:
Linux and macOS:
cat AkaLlama-llama3-70b-v0.1.Q6_K.*-of-00002.gguf > AkaLlama-llama3-70b-v0.1.Q6_K.gguf && rm AkaLlama-llama3-70b-v0.1.Q6_K.*-of-00002.gguf
cat AkaLlama-llama3-70b-v0.1.Q8_0.*-of-00002.gguf > AkaLlama-llama3-70b-v0.1.Q8_0.gguf && rm AkaLlama-llama3-70b-v0.1.Q8_0.*-of-00002.gguf
Windows command line:
COPY /B AkaLlama-llama3-70b-v0.1.Q6_K.00001-of-00002.gguf + AkaLlama-llama3-70b-v0.1.Q6_K.00002-of-00002.gguf AkaLlama-llama3-70b-v0.1.Q6_K.gguf
del AkaLlama-llama3-70b-v0.1.Q6_K.00001-of-00002.gguf AkaLlama-llama3-70b-v0.1.Q6_K.00002-of-00002.gguf
COPY /B AkaLlama-llama3-70b-v0.1.Q8_0.00001-of-00002.gguf + AkaLlama-llama3-70b-v0.1.Q8_0.00002-of-00002.gguf AkaLlama-llama3-70b-v0.1.Q8_0.gguf
del AkaLlama-llama3-70b-v0.1.Q8_0.00001-of-00002.gguf AkaLlama-llama3-70b-v0.1.Q8_0.00002-of-00002.gguf
| Model | #Parameter | Qunatized? | LogicKor* |
|---|---|---|---|
| AkaLlama-llama3-70b-v0.1-GGUF.Q4 | 70B | 4bit | 6.56 |
| AkaLlama-llama3-70b-v0.1-GGUF.Q8 | 70B | 8bit | 6.34 |
*mean over 3 random seeds
We trained AkaLlama using a preference learning alignment algorithm called Odds Ratio Preference Optimization (ORPO). Our training pipeline is almost identical to that of HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1, aside from minor hyperparameter changes. Please check out Huggingface's alignment handbook for further details, including the chat template.
Detailed descriptions regarding training data will be announced later.
You can find more examples at our project page
Although AKALlama-70B has significant potential, its responses can sometimes be inaccurate, biased, or misaligned, presenting risks if used without additional testing and refinement. Furthermore, the quality of the model's output is greatly influenced by the system prompt and decoding strategy. Changes in these areas could result in less precise outputs. Therefore, we strongly recommend handling our model with considerable caution.
If you find our model useful in your work, please cite the model as follows:
@misc{akallama,
author = {Chung, Jiwan and Jeon, Jaehyun and Kim, Saejin and Lim, Seungwon and Oh, Giyeong and Son, Yejin and Yu, Youngjae},
title = {AkaLlama: Yonsei University Large Language Model Project},
year = {2024},
publisher = {Hugging Face},
journal = {Hugging Face repository},
howpublished = {\url{https://huggingface.co/mirlab/AkaLlama-llama3-70b-v0.1}},
}
We look forward for your feedback and welcome collaboration on this exciting project!