Downloads · 30 days
110
1% of all-time downloads
RichardErkhov/h2oai_-_h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1-gguf
h2oai_-_h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
110
1% of all-time downloads
All-time downloads
7.9K
Public
Repo size
74.5 GB
Likes
0
Public
Click a slice to open those files.
.gguf87.5 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1 - GGUF
language:
This model was trained using H2O LLM Studio. Visit H2O LLM Studio to learn how to train your own large language models.
To use the model with the transformers library on a machine with GPUs, first make sure you have the transformers library installed.
pip install transformers==4.36.1
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="h2oai/h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# We use the HF Tokenizer chat template to format each message
# https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
{"role": "user", "content": "Why is drinking water so healthy?"},
]
prompt = pipe.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
res = pipe(
prompt,
max_new_tokens=256,
)
print(res[0]["generated_text"])
# <|system|>You are a friendly chatbot</s><|prompt|>Why is drinking water so healthy?</s><|answer|> Drinking water is healthy for several reasons: [...]
You can load the models using quantization by specifying load_in_8bit=True or load_in_4bit=True. Also, sharding on multiple GPUs is possible by setting device_map=auto.
MistralForCausalLM(
(model): MistralModel(
(embed_tokens): Embedding(32000, 4096, padding_idx=0)
(layers): ModuleList(
(0-31): 32 x MistralDecoderLayer(
(self_attn): MistralAttention(
(q_proj): Linear(in_features=4096, out_features=4096, bias=False)
(k_proj): Linear(in_features=4096, out_features=1024, bias=False)
(v_proj): Linear(in_features=4096, out_features=1024, bias=False)
(o_proj): Linear(in_features=4096, out_features=4096, bias=False)
(rotary_emb): MistralRotaryEmbedding()
)
(mlp): MistralMLP(
(gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
(up_proj): Linear(in_features=4096, out_features=14336, bias=False)
(down_proj): Linear(in_features=14336, out_features=4096, bias=False)
(act_fn): SiLUActivation()
)
(input_layernorm): MistralRMSNorm()
(post_attention_layernorm): MistralRMSNorm()
)
)
(norm): MistralRMSNorm()
)
(lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.
By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.