Downloads · 30 days
53
20% of all-time downloads
gobean/autotrain-mixtral-8x7b-orpo-v1-GGUF
autotrain-mixtral-8x7b-orpo-v1-GGUF is a text generation model from gobean. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
gobean: quants to q40, q50, q80 - the favorites for Mixtrals. Manually set EOS token to 2, llama.cpp has a bug.
Downloads · 30 days
53
20% of all-time downloads
All-time downloads
266
Public
Repo size
108 GB
Likes
0
Public
Click a slice to open those files.
.gguf108 GB · 100%
From the Hugging Face model README
gobean: quants to q4_0, q5_0, q8_0 - the favorites for Mixtrals. Manually set EOS token to 2, llama.cpp has a bug.
https://huggingface.co/abhishek/autotrain-mixtral-8x7b-orpo-v1
This model was trained using AutoTrain. For more information, please visit AutoTrain.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = "PATH_TO_THIS_REPO"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
model_path,
device_map="auto",
torch_dtype='auto'
).eval()
# Prompt content: "hi"
messages = [
{"role": "user", "content": "hi"}
]
input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt')
output_ids = model.generate(input_ids.to('cuda'))
response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
# Model response: "Hello! How can I assist you today?"
print(response)