Downloads · 30 days
16
4% of all-time downloads
Koshkasa/sophosympatheia_Magistry-24B-v1.1-IQ4_KS.gguf
sophosympatheia_Magistry-24B-v1.1-IQ4_KS.gguf is a text generation model from Koshkasa. Use it when you need the model to write or continue text. It is set up for ik_llama.cpp. The card lists the license as other.
The goal: Make a medium quality quant of sophosympatheia/Magistry-24B-v1.1 using SOTA quant types from ikllama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead. The resul…
Downloads · 30 days
16
4% of all-time downloads
All-time downloads
386
Public
Repo size
13.6 GB
Likes
0
Public
Click a slice to open those files.
.gguf13.6 GB · 100%
From the Hugging Face model README
The goal: Make a medium quality quant of sophosympatheia/Magistry-24B-v1.1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.
The result: Mixed precision quantization of sophosympatheia/Magistry-24B-v1.1
quantized with ik_llama.cpp build: 9d07d868
incompatible with mainline llama.cpp
| Layer | Dims | Dims | Quant |
|---|---|---|---|
| token_embd | 5120 | 131072.0 | iq4_k |
| blk | 40 | ||
| attn_k | 5120 | 1024 | iq6_k |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 4096 | iq6_k |
| attn_v | 5120 | 1024 | iq6_k |
| ffn_down | 32768 | 5120 | iq4_k |
| ffn_gate | 5120 | 32768 | iq4_ks |
| ffn_norm | 5120 | 1 | f32 |
| ffn_up | 5120 | 32768 | iq4_ks |
| attn_output | 4096 | 5120 | iq6_k |
| output | 5120 | 131072 | iq6_k |
| output_norm | 5120 | 1 | f32 |
MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
sophosympatheia - for the merge effort.
Everyone whose finetunes were included in the merge!
bartowski - for the imatrix + the myriad of quants we all benefit from.