Downloads · 30 days
66
24% of all-time downloads
Koshkasa/DarkArtsForge_Zepar-24B-v1_IQ3_KT_attn
DarkArtsForge_Zepar-24B-v1_IQ3_KT_attn is a text generation model from Koshkasa. Use it when you need the model to write or continue text. It is set up for ik_llama.cpp. The card lists the license as apache-2.0.
This is a mixed quantization of DarkArtsForge/Zepar-24B-v1 leveraging ikllama.cpp trellis quantization as well as IQK quants, created to test various approaches to trellis quantization of medium weight models. I also…
Downloads · 30 days
66
24% of all-time downloads
All-time downloads
270
Public
Repo size
58.1 GB
Likes
0
Public
Click a slice to open those files.
.gguf58.1 GB · 100%
From the Hugging Face model README
This is a mixed quantization of DarkArtsForge/Zepar-24B-v1 leveraging ik_llama.cpp trellis quantization as well as IQ_K quants, created to test various approaches to trellis quantization of medium weight models.
I also included the f16 GGUF for quicker quantization, as well as the imatrix I generated and used.
iq3_kt: ffn_gate, ffn_up (80)
iq4_kt: ffn_down, token_embd (41)
iq5_k: attn_q, attn_v, attn_k, attn_output (160)
iq6_k: output (1)
imatrix created based on calibration data by bartowski
quantized with ik_llama.cpp build: 9d07d868
incompatible with mainline llama.cpp
only tested with ik_llama.cpp (though I expect croco.cpp to work as well?)
I wanted to run the model and there were no quants at the time I started baking, simple as. While we're at it, I tried testing a mixed trellis quantization of ffn, applying more precision to ffn_down. The massive byte budget savings allowed to quant attention up, and the resulting file is still small. KVO with 65k ctx in tolerable cache quantization fits even with system overhead. Why do that? Idk, but it's fun.
-Is it worth it?
No idea. I made it for personal use and as an experiment. It didn't loose too much brain based on quick benchmarks (winogrande = 78.5320, hellaswag(s=42) = 329/400). It loads (yay?), works with my preferred sampler settings (yay?), and outputs actual text (yay?). A quick 32k ctx run showed it remembers details from before and can write coherently about them. I'm yet to test it thoroughly in RP. WYSIWYG.
-Possible 12G VRAM full fit with KVO?
Probably. A mixed IQ3_KT ffn / IQ4_KT attn would likely fit with some context headroom. By my observations, it remains smart enough to be actually suprisingly usable, while keeping the bulk weight ffn tensors at 3.125 bpw. I have no use for such a quant, but I can bake one if someone needs it
MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
DarkArtsForge/Naphula - for the effort of designing and producing the merge. And for reviving my interest in Mistral 24b.
+++ Everyone whose finetunes were included in the merge!
bartowski - for the calibration data + the myriad of quants we all benefit from.
imatrix generated on an intermediary q8_0 quant. I don't eat RAM for breakfast.
My only contribution is compute. This is neither my merge, nor my calibration data. Have fun.