Downloads · 30 days
782
5% of all-time downloads
duyntnet/openchat-3.6-8b-20240522-imatrix-GGUF
openchat-3.6-8b-20240522-imatrix-GGUF is a text generation model from duyntnet. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Quantizations of https://huggingface.co/openchat/openchat-3.6-8b-20240522
Downloads · 30 days
782
5% of all-time downloads
All-time downloads
14.3K
Public
Repo size
114 GB
Likes
1
Public
Click a slice to open those files.
.gguf114 GB · 100%
From the Hugging Face model README
Quantizations of https://huggingface.co/openchat/openchat-3.6-8b-20240522
💡 Default Mode: Best for coding, chat and general tasks
GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hi<|end_of_turn|>GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant:
⚠️ Notice: Remember to set <|end_of_turn|> as end of generation token.
The default template is also available as the integrated tokenizer.chat_template, which can be used instead of manually specifying the template:
messages = [
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hi"},
{"role": "user", "content": "How are you today?"}
]
tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "openchat/openchat-3.6-8b-20240522"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
messages = [
{"role": "user", "content": "Explain how large language models work in detail."},
]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(input_ids,
do_sample=True,
temperature=0.5,
max_new_tokens=1024
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))