Downloads · 30 days
10
20% of all-time downloads
RichardErkhov/h2oai_-_h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1-4bits
h2oai_-_h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1-4bits is a text generation model from RichardErkhov. Use it when you need the model to write or continue text. It is set up for transformers.
Downloads · 30 days
10
20% of all-time downloads
All-time downloads
49
Public
Parameters
7.4B
4.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors4.5 GB · 100%
How the weights are stored.
U87B · 95%
From the Hugging Face model README
Quantization made by Richard Erkhov.
h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1 - bnb 4bits
language:
This model was trained using H2O LLM Studio. Visit H2O LLM Studio to learn how to train your own large language models.
To use the model with the transformers library on a machine with GPUs, first make sure you have the transformers library installed.
pip install transformers==4.36.1
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="h2oai/h2ogpt-gm-7b-mistral-chat-sft-dpo-rag-v1",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# We use the HF Tokenizer chat template to format each message
# https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
{"role": "user", "content": "Why is drinking water so healthy?"},
]
prompt = pipe.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
res = pipe(
prompt,
max_new_tokens=256,
)
print(res[0]["generated_text"])
# <|system|>You are a friendly chatbot</s><|prompt|>Why is drinking water so healthy?</s><|answer|> Drinking water is healthy for several reasons: [...]
You can load the models using quantization by specifying load_in_8bit=True or load_in_4bit=True. Also, sharding on multiple GPUs is possible by setting device_map=auto.
MistralForCausalLM(
(model): MistralModel(
(embed_tokens): Embedding(32000, 4096, padding_idx=0)
(layers): ModuleList(
(0-31): 32 x MistralDecoderLayer(
(self_attn): MistralAttention(
(q_proj): Linear(in_features=4096, out_features=4096, bias=False)
(k_proj): Linear(in_features=4096, out_features=1024, bias=False)
(v_proj): Linear(in_features=4096, out_features=1024, bias=False)
(o_proj): Linear(in_features=4096, out_features=4096, bias=False)
(rotary_emb): MistralRotaryEmbedding()
)
(mlp): MistralMLP(
(gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
(up_proj): Linear(in_features=4096, out_features=14336, bias=False)
(down_proj): Linear(in_features=14336, out_features=4096, bias=False)
(act_fn): SiLUActivation()
)
(input_layernorm): MistralRMSNorm()
(post_attention_layernorm): MistralRMSNorm()
)
)
(norm): MistralRMSNorm()
)
(lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.
By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.