Hy4-preview-GGUF

Hy4-preview GGUF

GGUF quantizations of tencent/Hy4-preview, made with llama.cpp.

Chinese version: README_zh.md

780B total parameters, 78 layers. DeepSeek-style MLA attention (q-LoRA rank 2048, kv-LoRA rank 512) with a gated output path and learnable attention sinks, wrapped in iHC hyper-connections on every attention and MLP block. 256 routed MoE experts with top-8 routing plus one shared expert. DSA sparse attention: 21 of the 78 layers own a full indexer, the rest share the preceding full layer's top-k. Vocabulary 120,832.

Quantized from the official BF16 weights. Every tier is imatrix-calibrated and ships as 39 shards.

Quantizations

The master row is not a file in this repo. It is listed so the numbers above have a reference point — the 1.4 TiB BF16 master was actually loaded and measured on the same machine, so every tier is compared against a real baseline rather than an estimate.

Pick IQ4_XS unless you cannot fit it. It keeps the same top-1 token as the BF16 master on 90 % of positions and its KL divergence is 4.4× lower than the next tier down. Q2_K is the balanced choice and the best option for CPU-only inference. IQ2_XS exists for when the footprint is hard-capped.

What the imatrix is worth

Every tier here is calibrated with an importance matrix. To quantify what that is worth, Q2_K was built twice — once with the imatrix and once without — at byte-identical size (276,118 MiB both times), so the only variable is the calibration:

Measured against this repo's own size-quality curve, that 24 % KL reduction is worth +0.257 BPW — i.e. reaching the same quality without an imatrix would need a file 8.5 % larger (about 23 GiB more on the Q2_K tier). The imatrix itself costs nothing in the shipped weights.

Where the gain lands, by KL percentile:

The improvement is concentrated in the body of the distribution, not the tail: an imatrix makes the typical token markedly closer to the original model, while the rare catastrophic divergences are a property of low-bit quantization that calibration cannot undo. This is exactly what per-channel importance weighting is expected to do.

Calibration corpus: 476 KiB of real-world text, 220 chunks at n_ctx=512, run over the full BF16 master (6.7 h on 8× H100).

What is protected

The routed experts are 96.5 % of this model's weights, so everything else can be kept at high precision for almost no size cost. It is kept:

Usage

# Point at the first shard; llama.cpp finds the rest on its own.
llama-cli -m Hy4-preview-IQ4_XS-00001-of-00039.gguf

Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself, and an explicit -ngl — including 0 — aborts that fitting and can fail to allocate compute buffers even when the weights would have fit.

Every tier ships as 39 shards. Download all 39 into one directory — you only ever name -00001-of-00039 on the command line.

Requires a llama.cpp build with hy_v4 support (merged upstream 2026-09-04, PR #28127). Earlier builds will not load these files.

About the numbers

wikitext-2 test, n_ctx=512, 655 chunks (~335 K tokens), every tier through the exact same command on 8× H100 80GB. KL divergence and top-1 agreement are computed against logits saved from the BF16 master over 200 chunks.

These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make absolute values meaningless across setups.

Two notes on why the error bars here are small enough to be useful: the PPL runs use 655 chunks rather than the handful common in quantization write-ups, and KL divergence is reported alongside PPL because perplexity is a mean statistic that hides where a quantized model actually breaks. The percentile table above is only readable because both were done.

License

Apache-2.0, inherited from tencent/Hy4-preview — see LICENSE for terms. Quantized by qtum.