Qwen3.8-Flash-Next-APEX-GGUF

Qwen3.8-Flash-Next APEX (Custom Optimized Layout)

This repository provides optimized, high-fidelity GGUF quantization files for Qwen3.8-Flash-Next (the 125B/177B Mixture-of-Experts architecture featuring GDN + Qwen Sparse Attention).

WORKS WITH GUANACO in ~24GB, around 4 tokens per second on a Framework 13 AI 340.

🌟 Key Improvements over Upstream

• Enhanced Chat Template: Bundles a refined, custom Jinja chat template that drastically outperforms official upstream Qwen implementations on multi-turn logic, strict JSON schemas, and complex parallel tool calling.
• Clean Hub Integration: The legacy chat_template field has been entirely stripped from tokenizer_config.json to avoid client duplication conflicts, fully embracing modern embedded GGUF tokenizer formatting.

---

📦 Quantization Breakdown

GGUF utilizes a modern mixed-precision strategy. While smaller, ultra-sensitive components default to unquantized precision to minimize perplexity loss, the largest layers are aggressively stepped down.

---

⚡ Running 78GB Models on Consumer Hardware (Guanaco Disk Streaming)

Thanks to the Guanaco disk-streaming patchset, you can comfortably run this massive model on consumer laptops (like a Framework 13 with 48GB unified UMA RAM) at 5–6 tokens/sec for generation, and significantly faster speeds for prompt evaluation. That red mark on the quant can be safely ignored.

To run the Q6_K model efficiently using io_uring and streaming memory mapping, use the following llama.cpp execution parameters:

LLAMA_ARG_GUANACO_IO_URING=1 llama-server -hf Atomic-Germ/Qwen3.8-Flash-Next-APEX-GGUF:Q6_K --load-mode streaming -c 128000

🧠 The Role of the Importance Matrix (imatrix) in Disk Streaming

Unlike standard GGUF repositories where the imatrix.dat file is purely used for static calibration during quantization, this repo's imatrix serves a dual purpose optimized for Guanaco:

• Herd-Cache Optimization: The embedded matrix signals the Guanaco herd-cache loader exactly which high-frequency MoE experts should be permanently pinned in RAM from the moment the model boots.
• Minimized Latency: By keeping the most foundational experts resident in host memory (fitting perfectly within tighter allocations like a 48GB UMA laptop profile), disk swapping is strictly reserved for sparse, niche experts.

If your specific operational use case aligns with general multi-turn instruction-following and tool-use, loading the .dat file alongside the GGUF will yield a massive latency improvement.

Where the bits went

With half the model locked, the only budget left to allocate is ffn_gate_exps and ffn_up_exps, 80.5 B parameters between them. That is where these files differ from a stock recipe, in two ways:

Layer position. Earlier measurement on Qwen3.8-27B found FFN layers at the edges of the stack 2.63x more sensitive per byte than those in the middle, so the first and last few layers keep more bits and the middle is pushed hardest.

Gate and up are not the same. The same sweep measured ffn_up at 0.00806 dKL/GB against ffn_gate at 0.00596, so up is about 1.35x dearer. It is held one step above gate throughout. Shelf recipes give the two identical types.

Everything else, the attention tensors, output, shared experts and the hyper-connection heads, is 2.8% of the model combined. Pinning all of it high costs about 2 GB, so it is pinned high rather than economised on.

What the quantization costs

Perplexity against the BF16 weights, same text and same 4096 token context, six chunks each:

For 4.5x compression against a 354 GB original, with 45% of the model at roughly 1.6 bits per weight in Nano, that is a good result.

Two caveats, because they matter. The text is the same corpus the importance matrix was built from, which flatters both quants; a held-out set would be the fair test. And perplexity is a blunt instrument, which is why the sensitivity work behind these allocations used KL divergence instead. Read these as evidence the files are sound, not as a claim that they beat any particular alternative.

A KL comparison against BF16 is not cheap here for a specific reason: the logits file stores about 497 KB per token at this 248320 token vocabulary, so it runs to terabytes at long context.

Limitations

Throughput is not measured. Different quant types select different kernels, so speed is a separate question from quality per byte.

The three sizes are close together because the floor forces it. There is no Q4 or Q6 tier here: above roughly 80 GB the movable budget relaxes toward Q4, where a flat recipe is already near optimal, so a larger tier from us would be the same file you can get anywhere.

One measurement worth sharing

The n-gram table is the cheapest place in this model to take bits away, and that is not obvious.

Quantizing it at IQ4_NL instead of Q8_0 saves 25.6 GB and costs a KL divergence of 0.0324, a perplexity rise of 0.71%, and 6% of top-1 token predictions. That works out to 0.00126 KL per GB, which is about half the price of the cheapest tensor group measured on Qwen3.8-27B and roughly 28 times cheaper than the dearest. Measured on two builds differing in exactly that one tensor, scored over 100 chunks of held-out wikitext-2.

So every tier here keeps the table at IQ4_NL. Other providers move it to Q8_0 at their Q5 and Q6 tiers, which spends 25.6 GB on the one tensor that gives back the least.

🧩 Swapping and Customizing the Herd-Cache Matrix

• Default Matrix: The included imatrix gguf is sourced from the mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF calibration run. It serves as an excellent baseline for general multi-turn dialogue, general reasoning, and everyday assistant tasks.
• Fully Modular: Because of the architecture, you can swap this file out for any existing imatrix generated for a Qwen 3.8 Flash Next / Qwen4 model.

If you have a specialized importance matrix calibrated on a specific domain (e.g., heavy coding datasets, specialized math, or creative writing), simply rename it to Qwen3.8-Flash-Next-imatrix.dat and use it at boot. The Guanaco herd-cache loader will automatically adapt, pinning a completely different subset of domain-specific experts into your RAM allocation.

Thanks to @mudler; here are their things:

<!-- apex-banner-v2 --> <div style="background-color: #f59e0b; color: white; padding: 20px; border-radius: 10px; text-align: center; margin: 20px 0;"> <a href="https://www.patreon.com/cw/mudler" style="color: white; text-decoration: underline;">🎉 Patreon (Monthly)</a> &nbsp;|&nbsp; <a href="https://www.buymeacoffee.com/mudler" style="color: white; text-decoration: underline;">☕ Buy Me a Coffee</a> &nbsp;|&nbsp; <a href="https://github.com/sponsors/mudler" style="color: white; text-decoration: underline;">⭐ GitHub Sponsors</a> </p> </div>