Downloads · 30 days
50
7% of all-time downloads
Koshkasa/TheDrummer_Skyfall-31B-v4.2_IQ3_KT-GGUF
TheDrummer_Skyfall-31B-v4.2_IQ3_KT-GGUF is a text generation model from Koshkasa. Use it when you need the model to write or continue text. It is set up for ik_llama.cpp. The card lists the license as apache-2.0.
Skyfall is a heavy hitter for mainline consumer GPUs. I used to run it at IQ4XS on 16 GB with partial offload, which was way too slow. IQ3M was tolerable, but restrictive of KVO. IQ3XXS was permissive of KVO, but too…
Downloads · 30 days
50
7% of all-time downloads
All-time downloads
754
Public
Repo size
12.7 GB
Likes
1
Public
Click a slice to open those files.
.gguf12.7 GB · 100%
From the Hugging Face model README
Skyfall is a heavy hitter for mainline consumer GPUs. I used to run it at IQ4_XS on 16 GB with partial offload, which was way too slow.
IQ3_M was tolerable, but restrictive of KVO. IQ3_XXS was permissive of KVO, but too dumbed down.
This is basically a ik-llama.cpp answer to "how do I fit this model into 16 gb with KVO without exacerbating brain damage"
The goals: at least not dumber than IQ3_M, at least 32k context in VRAM, at least at Q5_0 K and Q4_1 V cache, with system overhead accounted for.
The result: a mainly IQ3_KT quantization of TheDrummer/Skyfall-31B-v4.2
iq3_s: token_embd (1)
iq3_kt: attn_k, attn_q, ffn_up, ffn_gate, ffn_down (270)
iq4_k: attn_output, attn_v, output (109)
imatrix by bartowski
quantized with ik_llama.cpp build: 9d07d868
incompatible with mainline llama.cpp
only tested with ik_llama.cpp (though I expect croco.cpp to work as well)
IQ3_M by bartowski:
winogrande: 74.9013 ± 1.2186 (949/1267)
hellaswag (n=400, s=42): 82.25%
with K q5_0, V q4_1, b=512, amb=64, ngl=999, 5 runs:
pp512: 899.37 ± 36.62
pp32768: 658.53 ± 2.11
tg128: 25.09 0± 0.39
tg512: 22.00 ± 0.02
size: 14314095296 bytes
this IQ3_KT:
winogrande: 75.5328 ± 1.2082 (957/1267) (+0.84%)
hellaswag (n=400, s=42): 82.75% (+0.61%)
with K q5_0, V q4_1, b=512, amb=64, ngl=999, 5 runs:
pp512: 975.41 ± 32.69 (+8.45%)
pp32768: 741.10 ± 7.88 (+12.54%)
tg128: 23.16 ± 0.08 (-7.69%)
tg512: 22.53 ± 0.15 (+2.5%)
size: 12658394016 bytes (11.57% smaller) (-1578.99 MB)
-Is it smarter?
Not in a statistically significant way, but it's not dumber.
-Is it slower?
No. It's actually faster in attn, which is not surprising, given less quality on attn layers in comparison to IQ3_M by bartowski.
-Is it smaller?
Hell yeah. -1579 MB puts it in IQ3_XXS territory by size
-Is it optimal?
No. I have a suspicion lowered attn quants will hurt it over long ctx (to be tested). Could try squeezing ffn_up in favor of k- and q- attn.
The primary candidate directly touching residuals is ffn_down which I can't quant up meaningfully (IQ3_KT is IQ_3M-ish in quality, going above 4bit bloats the gguf hard, failing the ctx goal.
One quantization takes me about 2.5 hrs on my CPU, so I can't freely test every hypothesis quickly.
-Is it worth it?
No idea. I made it for personal use to squeeze some extra context in without losing brainpower. I already had to resort to 3 bits. WYSIWYG.
-Possible 12G VRAM full fit with KVO?
Unlikely. A mixed IQ2_KT ffn / IQ3_KT attn quant was around 8.8 GB, but it scored significantly worse on hellaswag and winogrande. I didn't go for testing it in RP, since it was overkill compression for my purpose.
-Why Q5_0/Q4_1?
Reasons
MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
TheDrummer - you cooked a legendary one.
bartowski - for the imatrix + the myriad of quants we all benefit from.