Downloads · 30 days
15.1K
7% of all-time downloads
legraphista/openchat-3.6-8b-20240522-IMat-GGUF
openchat-3.6-8b-20240522-IMat-GGUF is a text generation model from legraphista. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as llama3.
Llama.cpp imatrix quantization of openchat/openchat-3.6-8b-20240522
Downloads · 30 days
15.1K
7% of all-time downloads
All-time downloads
221K
Public
Repo size
125 GB
Likes
1
Public
Click a slice to open those files.
.gguf125 GB · 100%
From the Hugging Face model README
Llama.cpp imatrix quantization of openchat/openchat-3.6-8b-20240522
Original Model: openchat/openchat-3.6-8b-20240522
Original dtype: BF16 (bfloat16)
Quantized by: llama.cpp b3006
IMatrix dataset: here
Status: ✅ Available
Link: here
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| openchat-3.6-8b-20240522.Q8_0.gguf | Q8_0 | 8.54GB | ✅ Available | ⚪ Static | 📦 No |
| openchat-3.6-8b-20240522.Q6_K.gguf | Q6_K | 6.60GB | ✅ Available | ⚪ Static | 📦 No |
| openchat-3.6-8b-20240522.Q4_K.gguf | Q4_K | 4.92GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.Q3_K.gguf | Q3_K | 4.02GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.Q2_K.gguf | Q2_K | 3.18GB | ✅ Available | 🟢 IMatrix | 📦 No |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| openchat-3.6-8b-20240522.FP16.gguf | F16 | 16.07GB | ✅ Available | ⚪ Static | 📦 No |
| openchat-3.6-8b-20240522.BF16.gguf | BF16 | 16.07GB | ✅ Available | ⚪ Static | 📦 No |
| openchat-3.6-8b-20240522.Q5_K.gguf | Q5_K | 5.73GB | ✅ Available | ⚪ Static | 📦 No |
| openchat-3.6-8b-20240522.Q5_K_S.gguf | Q5_K_S | 5.60GB | ✅ Available | ⚪ Static | 📦 No |
| openchat-3.6-8b-20240522.Q4_K_S.gguf | Q4_K_S | 4.69GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.Q3_K_L.gguf | Q3_K_L | 4.32GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.Q3_K_S.gguf | Q3_K_S | 3.66GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.Q2_K_S.gguf | Q2_K_S | 2.99GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ4_NL.gguf | IQ4_NL | 4.68GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ4_XS.gguf | IQ4_XS | 4.45GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ3_M.gguf | IQ3_M | 3.78GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ3_S.gguf | IQ3_S | 3.68GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ3_XS.gguf | IQ3_XS | 3.52GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ3_XXS.gguf | IQ3_XXS | 3.27GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ2_M.gguf | IQ2_M | 2.95GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ2_S.gguf | IQ2_S | 2.76GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ2_XS.gguf | IQ2_XS | 2.61GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ2_XXS.gguf | IQ2_XXS | 2.40GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ1_M.gguf | IQ1_M | 2.16GB | ✅ Available | 🟢 IMatrix | 📦 No |
| openchat-3.6-8b-20240522.IQ1_S.gguf | IQ1_S | 2.02GB | ✅ Available | 🟢 IMatrix | 📦 No |
If you do not have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Download the specific file you want:
huggingface-cli download legraphista/openchat-3.6-8b-20240522-IMat-GGUF --include "openchat-3.6-8b-20240522.Q8_0.gguf" --local-dir ./
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/openchat-3.6-8b-20240522-IMat-GGUF --include "openchat-3.6-8b-20240522.Q8_0/*" --local-dir openchat-3.6-8b-20240522.Q8_0
# see FAQ for merging GGUF's
<|begin_of_text|><|start_header_id|>GPT4 Correct User<|end_header_id|>
Can you provide ways to eat combinations of bananas and dragonfruits?<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>
Sure! Here are some ways to eat bananas and dragonfruits together:
1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey.<|eot_id|><|start_header_id|>GPT4 Correct User<|end_header_id|>
What about solving an 2x + 3 = 7 equation?<|eot_id|>
<|begin_of_text|><|start_header_id|>System<|end_header_id|>
You are a helpful AI.<|eot_id|><|start_header_id|>GPT4 Correct User<|end_header_id|>
Can you provide ways to eat combinations of bananas and dragonfruits?<|eot_id|><|start_header_id|>GPT4 Correct Assistant<|end_header_id|>
Sure! Here are some ways to eat bananas and dragonfruits together:
1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey.<|eot_id|><|start_header_id|>GPT4 Correct User<|end_header_id|>
What about solving an 2x + 3 = 7 equation?<|eot_id|>
llama.cpp/main -m openchat-3.6-8b-20240522.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
gguf-split available
gguf-split, navigate to https://github.com/ggerganov/llama.cpp/releasesgguf-splitopenchat-3.6-8b-20240522.Q8_0)gguf-split --merge openchat-3.6-8b-20240522.Q8_0/openchat-3.6-8b-20240522.Q8_0-00001-of-XXXXX.gguf openchat-3.6-8b-20240522.Q8_0.gguf
gguf-split to the first chunk of the split.Got a suggestion? Ping me @legraphista!