Downloads · 30 days
89
7% of all-time downloads
RichardErkhov/karakuri-ai_-_karakuri-lm-8x7b-chat-v0.1-gguf
karakuri-ai_-_karakuri-lm-8x7b-chat-v0.1-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
89
7% of all-time downloads
All-time downloads
1.2K
Public
Repo size
530 GB
Likes
0
Public
Click a slice to open those files.
.gguf614 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
karakuri-lm-8x7b-chat-v0.1 - GGUF
library_name: transformers license: apache-2.0 datasets:
![]()
[email protected]The prompt format has been changed from KARAKURI LM 70B Chat v0.1. Please make sure to follow the correct format. Otherwise, the model will generate sub-optimal outputs.
We use the following prompt template of multi-turn conversation in the Mistral format, which includes an encoded string of multiple attribute values.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("karakuri-ai/karakuri-lm-8x7b-chat-v0.1")
messages = [
{"role": "system", "content": "System prompt"},
{"role": "user", "content": "User prompt"},
{"role": "assistant", "content": "Model response"},
{"role": "user", "content": "User prompt"},
]
tokenizer.apply_chat_template(messages, tokenize=False)
# <s>[INST] <<SYS>>
# System prompt
# <</SYS>>
#
# User prompt [ATTR] helpfulness: 4 correctness: 4 coherence: 4 complexity: 4 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]Model response</s>[INST] User prompt [ATTR] helpfulness: 4 correctness: 4 coherence: 4 complexity: 4 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]
The prompt template contains nine attributes. The first five are derived from HelpSteer, while the remaining four are derived from OASST2. The values are represented by integers ranging from 0 to 4, with 0 being the lowest and 4 being the highest.
If you want to change attribute values from the default values specified in the template, you can modify them to any values by adding the attribute values to the user messages:
messages = [
{"role": "user", "content": "User prompt", "helpfulness": 0, "complexity": 0},
]
tokenizer.apply_chat_template(messages, tokenize=False)
# <s>[INST] User prompt [ATTR] helpfulness: 0 correctness: 4 coherence: 4 complexity: 0 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"karakuri-ai/karakuri-lm-8x7b-chat-v0.1",
torch_dtype="auto",
device_map="auto",
)
messages = [
{
"role": "user",
"content": "週末に日帰りで東京に遊びに行こうと思っています。日帰りなので、短時間で回れるおすすめの観光プランを教えてください。",
},
]
input_ids = tokenizer.apply_chat_template(
messages,
return_tensors="pt",
).to(model.device)
outputs = model.generate(input_ids, max_new_tokens=512)
tokenizer.decode(outputs[0][input_ids.shape[-1]:])
| Model | # Active Params | Alignment | MT-Bench-jp |
|---|---|---|---|
| Qwen1.5 72B Chat | 72B | DPO | 8.19 |
| KARAKURI LM 8x7B Chat v0.1 | 13B | SteerLM | 7.54 |
| Command R+ | 104B | - | 7.31 |
| Mixtral 8x7B Instruct v0.1 | 13B | DPO | 7.24 |
| Llama 3 70B Instruct | 70B | RLHF | 7.13 |
| KARAKURI LM 70B Chat v0.1 | 70B | SteerLM | 6.43 |
| Llama 2 70B Chat | 70B | RLHF | 5.23 |
@misc{karakuri_lm_8x7b_chat_v01,
author = { {KARAKURI} {I}nc. },
title = { {KARAKURI} {LM} 8x7{B} {C}hat v0.1 },
year = { 2024 },
url = { https://huggingface.co/karakuri-ai/karakuri-lm-8x7b-chat-v0.1 },
publisher = { Hugging Face },
journal = { Hugging Face repository }
}