Vortex5_Shadow-Siren-26B-A4B-APEX-GGUF

What's that?

As denoted by its creators, APEX is "a novel MoE-aware mixed-precision quantization technique for llama.cpp". I suggest reading their information on the how and why of the method.

These two GGUFs are APEX quantizations of Vortex5/Shadow-Siren-26B-A4B, using EmanuelOverride's custom APEX recipe, HarmonicMinus, and a minor customization of that edit to cap shared ffn at Q6_K and replace Q4_K and Q4_0 (which was a fallback for ffn_down_exps from Q3_K) sparse expert quants with MXFP4. Both recipes use mradermacher's imatrix found here

To quote EmanuelOverride regarding the philosophy behind HarmonicMinus:

---

🎵 The Harmonic Series (Pulsed Attention)

Following on from the Harmonic Resonance branch, the Harmonic layouts further optimize layer contiguity for performance, by following strict precision tiers across all 30 layers (High/Medium/Low), while priorizing high-precision tiers for Global Attention (GA) layers, preventing aggressive compression in order to anchor deep-context tracking.

• i-Harmonic.Minus: 14,369.74 MiB (4.78 BPW) — High-performance configuration. Condenses the architecture into 3 strict tiers (5/15/10). Shifts high-precision blocks to coincide with global attention pulses.

---

My modification, aptly named HarmonicMinusMinus, retains most of the original recipe and adheres to the same philosophy, sacrificing some precision on sparse experts FFN layers for extra VRAM savings, while retaining global attention-focused attn distribution. A secondary effect is minor speedup on GPUs with native fp4 support. The HarmonicMinusMinus modification APEX recipe is provided.

Benchmarks

pp32768/tg512 tests were run at q4_0/q4_0 cache quantization to ensure both models fully fit in VRAM with KV cache on my GPU.

Conclusion

WYSIWYG. MXFP4 is lower precision than Q4_K and can underrepresent outliers. For certain ffn_down_exps it may have had an impact that had not been identified during my surface testing. Feedback welcome.

Disclosure

My only contribution is compute and a little tinkering with an already existing custom APEX recipe. This is neither my merge, nor my quantization methodology. Have fun.

Cheers

Google - the base model.
Vortex5 - for the merge effort.
Everyone whose finetunes were included in the merge!
mudler - for the APEX quantization methodology.
EmanuelOverride - for the custom APEX experiments and the base recipe.
mradermacher - for the imatrix + myriads of quants we all benefit from.