hermeo-7b-awq

Hermes + Leo = Hermeo

Hermeo-7B AWQ quantized

A quantized version of the malteos German-English language model hermeo-7b merged from DPOpenHermes-7B-v2 and leo-mistral-hessianai-7b-chat using mergekit. Both base models are fine-tuned versions of Mistral-7B-v0.1.

Model details

• Quantized from: hermeo-7b
• Merged from: leo-mistral-hessianai-7b-chat and DPOpenHermes-7B-v2
• Model type: Causal decoder-only transformer language model
• Languages: English and German
• License: Apache 2.0

How to use

Requires:

• Transformers from commit 72958fcd3c98a7afdc61f953aa58c544ebda2f79
• AutoAWQ from commit 1c5ccc791fa2cb0697db3b4070df1813f1736208.

pip3 install git+https://github.com/huggingface/transformers.git@72958fcd3c98a7afdc61f953aa58c544ebda2f79
 
pip3 install git+https://github.com/casper-hansen/AutoAWQ.git@1c5ccc791fa2cb0697db3b4070df1813f1736208

You currently can use this model using AutoAWQForCausalLM. Inference should be possible with transformers pipeline as well, but is not yet supported by AutoAWQ.

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_name_or_path = "mayflowergmbh/hermeo-7b-awq"
model = AutoAWQForCausalLM.from_quantized(model_name_or_path, fuse_layers=True,
                                          trust_remote_code=False, safetensors=True)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=False)

prompt = 'Hallo, Ich bin ein Sprachmodell,'
tokens = tokenizer(
    prompt,
    return_tensors='pt'
).input_ids.cuda()

generation_output = model.generate(
        tokens,
        do_sample=True,
        max_new_tokens=512
)

print(tokenizer.decode(generation_output[0]))

Acknowledgements

• This model release is heavily inspired by Weyaxi/OpenHermes-2.5-neural-chat-v3-2-Slerp
• Thanks to the authors of the base models: Mistral, LAION, HessianAI, Open Access AI Collective, @teknium, @bjoernp
• The German evaluation datasets and scripts from @bjoernp were used.
• The computing resources from DFKI's PEGASUS cluster were used for the evaluation.

Evaluation

The evaluation methdology of the Open LLM Leaderboard is followed.

German benchmarks

English benchmarks

TBA

Prompting / Prompt Template

Prompt dialogue template (ChatML format):

"""
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
"""

The model input can contain multiple conversation turns between user and assistant, e.g.

<|im_start|>user
{prompt 1}<|im_end|>
<|im_start|>assistant
{reply 1}<|im_end|>
<|im_start|>user
{prompt 2}<|im_end|>
<|im_start|>assistant
(...)

License

Apache 2.0