Downloads · 30 days
33
5% of all-time downloads
gobean/autotrain-mixtral-8x7b-orpo-v2-GGUF
autotrain-mixtral-8x7b-orpo-v2-GGUF is a text generation model from gobean. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
gobean: quants at q40, q50, q80 since it's a Mixtral. Manually set EOS due to bug in llama.cpp.
Downloads · 30 days
33
5% of all-time downloads
All-time downloads
619
Public
Repo size
108 GB
Likes
0
Public
Click a slice to open those files.
.gguf108 GB · 100%
From the Hugging Face model README
gobean: quants at q4_0, q5_0, q8_0 since it's a Mixtral. Manually set EOS due to bug in llama.cpp.
https://huggingface.co/abhishek/autotrain-mixtral-8x7b-orpo-v2
This model was trained using AutoTrain. For more information, please visit AutoTrain.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = "PATH_TO_THIS_REPO"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
model_path,
device_map="auto",
torch_dtype='auto'
).eval()
# Prompt content: "hi"
messages = [
{"role": "user", "content": "hi"}
]
input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt')
output_ids = model.generate(input_ids.to('cuda'))
response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
# Model response: "Hello! How can I assist you today?"
print(response)