Downloads · 30 days
313
1% of all-time downloads
legraphista/Qwen2-57B-A14B-Instruct-IMat-GGUF
Qwen2-57B-A14B-Instruct-IMat-GGUF is a text generation model from legraphista. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
<br <div style="padding: 16px 32px; outline: 2px solid; border-radius: 10px; outline-color: red; margin: 12px" Currently investigating issue quantizing imatirx variants.
Downloads · 30 days
313
1% of all-time downloads
All-time downloads
35.5K
Public
Repo size
520 GB
Likes
2
Public
Click a slice to open those files.
.gguf418 GB · 100%
From the Hugging Face model README
For static quants, visit <a href="https://huggingface.co/legraphista/Qwen2-57B-A14B-Instruct-GGUF">legraphista/Qwen2-57B-A14B-Instruct-GGUF</a>
<pre> [ 12/ 479] blk.0.ffn_gate_exps.weight - [ 3584, 2560, 64, 1], type = f32, converting to q4_K .. ggml_validate_row_data: found nan value at block 1 ggml_validate_row_data: found nan value at block 0 ggml_validate_row_data: found nan value at block 0 ggml_validate_row_data: found nan value at block 0 ggml_validate_row_data: found nan value at block 14 </pre> </div>Llama.cpp imatrix quantization of Qwen/Qwen2-57B-A14B-Instruct
Original Model: Qwen/Qwen2-57B-A14B-Instruct
Original dtype: BF16 (bfloat16)
Quantized by: llama.cpp b3091
IMatrix dataset: here
Status: ✅ Available
Link: here
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| Qwen2-57B-A14B-Instruct.Q8_0/* | Q8_0 | 61.02GB | ✅ Available | ⚪ Static | ✂ Yes |
| Qwen2-57B-A14B-Instruct.Q6_K/* | Q6_K | 47.12GB | ✅ Available | ⚪ Static | ✂ Yes |
| Qwen2-57B-A14B-Instruct.Q4_K | Q4_K | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q3_K | Q3_K | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q2_K | Q2_K | - | ❌ Errored | 🟢 IMatrix | - |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| Qwen2-57B-A14B-Instruct.BF16/* | BF16 | 114.84GB | ✅ Available | ⚪ Static | ✂ Yes |
| Qwen2-57B-A14B-Instruct.FP16/* | F16 | 114.84GB | ✅ Available | ⚪ Static | ✂ Yes |
| Qwen2-57B-A14B-Instruct.Q8_0/* | Q8_0 | 61.02GB | ✅ Available | ⚪ Static | ✂ Yes |
| Qwen2-57B-A14B-Instruct.Q6_K/* | Q6_K | 47.12GB | ✅ Available | ⚪ Static | ✂ Yes |
| Qwen2-57B-A14B-Instruct.Q5_K.gguf | Q5_K | 40.80GB | ✅ Available | ⚪ Static | 📦 No |
| Qwen2-57B-A14B-Instruct.Q5_K_S.gguf | Q5_K_S | 39.57GB | ✅ Available | ⚪ Static | 📦 No |
| Qwen2-57B-A14B-Instruct.Q4_K | Q4_K | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q4_K_S | Q4_K_S | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ4_NL | IQ4_NL | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ4_XS | IQ4_XS | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q3_K | Q3_K | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q3_K_L | Q3_K_L | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q3_K_S | Q3_K_S | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ3_M | IQ3_M | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ3_S | IQ3_S | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ3_XS | IQ3_XS | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ3_XXS | IQ3_XXS | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q2_K | Q2_K | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.Q2_K_S | Q2_K_S | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ2_M | IQ2_M | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ2_S | IQ2_S | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ2_XS | IQ2_XS | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ2_XXS | IQ2_XXS | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ1_M | IQ1_M | - | ❌ Errored | 🟢 IMatrix | - |
| Qwen2-57B-A14B-Instruct.IQ1_S | IQ1_S | - | ❌ Errored | 🟢 IMatrix | - |
If you do not have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Download the specific file you want:
huggingface-cli download legraphista/Qwen2-57B-A14B-Instruct-IMat-GGUF --include "Qwen2-57B-A14B-Instruct.Q8_0.gguf" --local-dir ./
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/Qwen2-57B-A14B-Instruct-IMat-GGUF --include "Qwen2-57B-A14B-Instruct.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
<|im_start|>user
{next_user_prompt}<|im_end|>
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
<|im_start|>user
{next_user_prompt}<|im_end|>
llama.cpp/main -m Qwen2-57B-A14B-Instruct.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
gguf-split available
gguf-split, navigate to https://github.com/ggerganov/llama.cpp/releasesgguf-splitQwen2-57B-A14B-Instruct.Q8_0)gguf-split --merge Qwen2-57B-A14B-Instruct.Q8_0/Qwen2-57B-A14B-Instruct.Q8_0-00001-of-XXXXX.gguf Qwen2-57B-A14B-Instruct.Q8_0.gguf
gguf-split to the first chunk of the split.Got a suggestion? Ping me @legraphista!