Downloads · 30 days
23
10% of all-time downloads
hyperhuzaifa/Hy3-mix12-IQ2XXS-Q4KM-GGUF
Hy3-mix12-IQ2XXS-Q4KM-GGUF is a machine learning model from hyperhuzaifa. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
A mixed-precision Hy3 (295B-A21B MoE) GGUF: an IQ2XXS base with the expert tensors of its 12 most sensitive layers (68–79) replaced by Q4KM ones, spliced in at the byte level with no requantization.
Downloads · 30 days
23
10% of all-time downloads
All-time downloads
241
Public
Repo size
99.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf99.7 GB · 100%
From the Hugging Face model README
A mixed-precision Hy3 (295B-A21B MoE) GGUF: an IQ2_XXS base with the expert tensors of its 12 most sensitive layers (68–79) replaced by Q4_K_M ones, spliced in at the byte level with no requantization.
Result: −4.9% NLL versus the pure 2-bit base, for +14 GB.
| Base IQ2_XXS | This model | |
|---|---|---|
| Size | 79 GB | 92.8 GB (99,681,728,320 bytes) |
| Avg NLL vs bf16 reference | 0.7583 | 0.7211 (−4.911%) |
| Per-case wins (100 prompts) | 37 | 63 (0 ties, sign test p ≈ 0.01) |
| First-token agreement with bf16 | 67/100 | 69/100 |
This applies antirez's DeepSeek-V4 mixed-quant idea to Hy3. The insight: in a MoE, the expert tensors dominate the file, and layers differ a lot in how much they suffer from 2-bit quantization. So you keep 2-bit almost everywhere and spend bits only where they buy the most.
Because both parents were quantized from the same BF16 weights with the same tensor layout, donor expert tensors can be copied verbatim into the base file — a byte-level splice, no dequant/requant round trip, no quality loss beyond the donor's own.
Layer selection was driven by imatrix activation statistics (in_sum2 / counts) from the base model's own importance matrix. Sensitivity was near-monotonic with depth, so the top-12 came out as a contiguous block: layers 68–79.
What was replaced — 36 tensors total, 3 per layer × 12 layers:
| Tensor | Base type | Donor type |
|---|---|---|
blk.{68..79}.ffn_down_exps.weight | IQ2_XXS | Q5_K |
blk.{68..79}.ffn_gate_exps.weight | IQ2_XXS | Q4_K |
blk.{68..79}.ffn_up_exps.weight | IQ2_XXS | Q4_K |
Everything else — all other expert layers, attention (Q8_0), embeddings, router, and the MTP block — is untouched from the base.
Only ~24.7 GB was fetched from the 185 GB donor, using HTTP range requests over the exact byte spans of the 36 needed tensors. fetch_plan.json in this repo records those offsets, so the build is reproducible without downloading the full donor.
Quality was measured against the official bf16 model, not against another quant:
temperature=0, seed=42, producing reference continuations. Seeding matters — unseeded temp-0 sampling on that endpoint was not reproducible; seeded, it was byte-identical across repeats.Lower NLL is better. The −4.9% gap means this model's next-token distributions sit measurably closer to the bf16 original.
This is a ds4-lineage GGUF and it keeps its 80 MTP tensors (1298 tensors total). Mainline llama.cpp will refuse to load it with a tensor-count mismatch, because its Hy3 mapping does not consume the MTP block.
It runs in pulsar (SSD-streaming MoE engine), which is what it was built and scored on.
The shards here are standard gguf-split output; point your loader at shard 00001-of-00004.
On a 2×24 GB VRAM / 96 GB RAM box, this model decodes slower than the 2-bit base (~3.1 vs ~5.0 tok/s) — not because the math is heavier, but because the fatter Q4_K/Q5_K slabs push the hot working set past what the cache tiers hold. On a machine whose fast memory comfortably fits 93 GB, the gap should close. Budget accordingly: this is a quality upgrade, and on constrained memory you pay for it in throughput.
This is a derivative of Hy3 and inherits the upstream model's license terms; consult the original model repository before use.