Downloads · 30 days
15
3% of all-time downloads
auroter/Hy3-295B-A21B-NVFP4A16-GPTQ
Hy3-295B-A21B-NVFP4A16-GPTQ is a text generation model from auroter. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Downloads · 30 days
15
3% of all-time downloads
All-time downloads
567
Public
Parameters
174B
178 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors178 GB · 100%
How the weights are stored.
U8143B · 82%
From the Hugging Face model README
This is a mixed-precision quantization of tencent/Hy3 (295B-A21B MoE). It was built for one goal: a faster decode rate without sacrificing quality. Decode speed on a large model is set by memory bandwidth — how many bytes of weights each token must read — and NVFP4 was chosen exactly to cut those bytes. The official FP8 release reads ~19.6 GB per token; this build reads ~14.4 GB. The measured speed follows that arithmetic. It loads on stock vLLM, and the multi-token-prediction (MTP) draft head is preserved bit-for-bit, so speculative decoding works.
<sub>¹ MTP throughput requires a two-line vLLM patch, submitted upstream as PR #48767 — see decision 4. All speed numbers measured on 4× RTX PRO 6000 Blackwell (SM120), single stream.</sub>
45,504 matrices — 79 MoE layers × 192 experts × gate/up/down — at FP4-E2M1, group size 16, FP8-E4M3 scales.
Why 4-bit here. The experts hold ~96% of the weights, and the 8 of 192 that fire still hold ~60% of the parameters each token touches. Four-bit here does two things at once: the artifact fits in 178 GB instead of 557, and the per-token read drops by ~5 GB versus the FP8 release — which is the entire speed win. Weight bytes are the decode bottleneck; the experts are where the bytes are.
Why calibrated. Data-free rounding is not safe on this checkpoint. An earlier data-free (RTN) NVFP4 build of ours corrupted rare tokens under Tencent's recommended sampling (temperature 0.9, top_p 1.0): about one event per thousand characters, single words written half in English and half in Mandarin. That build was deleted; the rate is a production observation. GPTQ solves each matrix against real activations on the same 4-bit grid, and the defect class vanished: zero mid-word corruption in 101,400 generated tokens.
560 matrices — attention (q/k/v/o), shared experts, and the dense layer-0 MLP — at FP8, per-channel weight scales, dynamic per-token activation scales.
Why. These layers run on every token, so they set both the speed floor and the quality floor. A predecessor build with every linear layer at 4-bit cost +3.41% perplexity. Holding these layers at FP8 recovered quality to +2.35%, and the build still reads 5 GB/token less than the official FP8. Tencent keeps exactly these layers at 8-bit in both of their own quantized releases. Their sensitivity data and ours agree.
All 593 tensors of the draft layer, byte-identical to source. Embeddings, the output head, router gates, norms, and expert biases also stay BF16 — small, logit-adjacent tensors that are not worth any risk.
Why. A draft head earns its keep through acceptance rate, and under speculative decoding acceptance is speed: every accepted draft token is a token the target did not have to decode alone. The head stays at full 16-bit on the belief that any quantization there spends acceptance — and spending acceptance is spending speed — to save a few GB of disk that never justify it. The official 4-bit GGUF quantizes its draft head; this build keeps it whole.
Why. The head is a single layer trained for depth 1 —
num_nextn_predict_layers: 1in the base config. Position-0 acceptance is ~57%, and it is set by that training, not by precision: this build's full-BF16 head measures the same acceptance as the official FP8's partly-quantized one (56.9% vs 57.7%). Past position 0, acceptance collapses. Measured on this build, single-stream: depth 1 = 117.7 tok/s, depth 2 = 102.1, depth 3 = 109.6 — conditional acceptance falls to ~41% at position 1 and ~37% at position 2, and every depth past 1 loses to depth 1 (depth 2 lands below even plain decode). Depth 1 pays +8.5% over plain. Deeper pays less.
One engine fact. vLLM needs a two-line patch for MTP to be profitable on this model family.
HYV3MTPinvllm/model_executor/models/hy_v3_mtp.pylacks the@support_torch_compiledecorator thatDeepSeekMTPcarries, so the draft layer runs eager against a CUDA-graphed target, and one eager draft cycle costs about one full target step. Add the import and the decorator, matching DeepSeekMTP. Upstream: issue vllm-project/vllm#48766 and PR #48767 — the PR's two-line diff is the exact local patch to apply to your install until it merges. Plain decode needs no patch.
Why a fix is needed at all. The base checkpoint leaks Chinese synonyms into pure-English prose, at every precision. At semantic slots it places the Chinese word at rank 2–3 of the next-token distribution with double-digit probability.
Measured at a real trigger slot (mid-sentence English narrative, where the next word is "denial" or "denying"), forced-token logprobs, both arms served identically:
| Arm | logprob(否认) | ≈ probability | rank |
|---|---|---|---|
| Official FP8 | −0.98 | ~37% | 2 |
| This build | −1.66 | ~19% | 3 |
A second trigger repeats the pattern (FP8 −4.30, this build −6.09). The official FP8 leaks harder than this build at both measured slots. In long-form English generation this surfaces about twice per 100K tokens — rare, never zero.
Why the weights cannot fix it. The leak survives Tencent's own FP8. It is in the checkpoint's training, not in anyone's rounding. And a rank-2 token at 37% cannot be filtered by any top_p, top_k, or min_p setting.
Why this exact mechanism. The sampler can do what the weights cannot: ban every token containing a Chinese, Japanese, or Korean (CJK) script character, passed as
allowed_token_ids. Control tokens are exempt — their names carry fullwidth-bar punctuation from the CJK block, and banning them removes end-of-sequence, leaving the model unable to stop; the ban governs text, grammar tokens stay emittable. vLLM enforcesallowed_token_idson both the plain and the speculative-decoding sampling paths. A custom logits processor silently fails to bind to MTP-drafted tokens;bad_wordsre-tokenizes ~48K strings per request. The ~1 MB payload costs milliseconds locally. Assembling a CJK character from raw byte tokens would take several consecutive tokens the model gives negligible probability in English, so banning whole tokens closes the leak.
The kit, in cjk-ban/: generate-allowed-ids.py (model-agnostic — scans any tokenizer, bans Han, kana, Hangul, and CJK punctuation, keeps control tokens emittable) and hy3-allowed-token-ids.json (pregenerated for this model: 47,295 banned, 73,523 allowed, of 120,818). The fix is not specific to this build: it works with any quant of this model — or the unquantized release — on anything that honors allowed_token_ids.
import json
from openai import OpenAI
allowed = json.load(open("cjk-ban/hy3-allowed-token-ids.json"))["allowed_token_ids"]
client = OpenAI(base_url="http://localhost:30000/v1", api_key="unused")
out = client.chat.completions.create(
model="hy3-nvfp4",
messages=[{"role": "user", "content": "Write a scene where he denies it."}],
temperature=0.9,
extra_body={"allowed_token_ids": allowed},
)
Offline: SamplingParams(allowed_token_ids=allowed).
Verified live under MTP at the strongest measured trigger, production sampling: unbanned, 4 of 8 samples leak 否认. Banned, 0 of 8. Greedy decode picks " denial". Generation terminates normally under the ban — finish_reason stop, verified live on this build. If you want English output with zero Chinese characters — from this build or from any serving of this checkpoint — use this.
All numbers measured on 4× RTX PRO 6000 Blackwell: vLLM 0.24.0, TP4, kv-cache fp8, single-stream decode, against the official tencent/Hy3-FP8 served identically on the same machine. Single-stream is deliberate: this build is for a personal workstation, and a workstation serves one person at a time. Aggregate multi-stream throughput is a data-center metric.
| Config | Official FP8 | This build |
|---|---|---|
| Plain decode (tok/s) | 96.9 | 108.5 (+12.0%) |
| MTP depth 1 (tok/s)¹ | 105.7 | 117.7 (+11.4%) |
| MTP depth 2 (tok/s) | 87.2 | 102.1 |
| MTP depth 3 (tok/s) | — | 109.6 |
| Perplexity, wikitext-2 (66,055 tokens) | 4.8624 | 4.9768 (+2.35%) |
| Stray Chinese-character events at temp 0.9 / top_p 1.0 | 0 / 50,598 tokens | 2 / 101,400 tokens² |
| MTP position-0 acceptance | 57.7% | 56.9% |
<sub>² Both events are the same whole-word substitution documented under decision 5 — a base-model behavior the official FP8 exhibits more strongly at the measured trigger slots. The rates are statistically inseparable, and the ban kit makes the rate exactly zero. Zero mid-word corruption in either arm.</sub>
On the depth rows: the base model is trained for an MTP depth of exactly one — num_nextn_predict_layers: 1 in Tencent's config, one draft module trained to predict one step ahead. Higher depths re-run that module on its own output, which its training never covered, so acceptance collapses past position 0 and every depth above 1 measures slower than depth 1. That is a property of the checkpoint, not of this quantization. Run depth 1.
Capability benchmarks are not re-run here; see Tencent's model card.
This build is designed and tested on 4× RTX PRO 6000 Blackwell (96 GB each). Two DGX Spark units hold the weights in 256 GB of combined unified memory, and NVFP4 is native to their Blackwell silicon too. Two RTX PRO 6000 cards (192 GB) might load it with a much smaller context window. Neither two-device setup is tested here.
vllm serve /path/to/Hy3-NVFP4A16-GPTQ \
--tensor-parallel-size 4 \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--gpu-memory-utilization 0.90
With MTP (after the patch in decision 4):
vllm serve /path/to/Hy3-NVFP4A16-GPTQ \
--tensor-parallel-size 4 \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--gpu-memory-utilization 0.90 \
--speculative-config '{"method": "hy_v3_mtp", "num_speculative_tokens": 1, "draft_tensor_parallel_size": 1}'
VLLM_ENGINE_READY_TIMEOUT_S=1800) so the engine is not killed mid-compile.<think:opensource> … </think:opensource>) as single tokens, and the chat template opens the section inside the prompt, so completions usually carry only the closer. On vLLM 0.24.x, --reasoning-parser hy_v3 expects the bare pre-release <think> pair and returns empty content — serve those versions with no reasoning parser and split client-side: everything up to the last </think:…> closer is reasoning, the rest is the answer. From 0.25.0, vllm-project/vllm#47192 resolves the namespaced pair natively via the token_suffix field this artifact's tokenizer_config ships.Tencent's AngelSlim team published Q4_K_M GGUFs of this model on 2026-07-14. Their recipe independently confirms this precision map: every-token layers at 8-bit, routed experts at 4-bit class. The differences:
| AngelSlim Q4_K_M | This build | |
|---|---|---|
| MTP draft head | quantized (eh_proj q8_0) | BF16, byte-identical |
| Expert 4-bit method | data-free K-quant (no imatrix published) | GPTQ, Hessian-calibrated per expert |
| Weight format | ggml K-blocks | NVFP4 (E2M1 + FP8 scales), tensor-core native on SM120 |
| Engine | llama.cpp at a pinned commit with local patches | stock vLLM 0.24, continuous batching, 256K context |
| Size | 182–185 GB | 178 GB, draft head included |
The official 4-bit release holds some expert tensors above 4-bit — the down projections at 5-bit, and the last layers' at 6-bit. That is the release team's own sensitivity signal. A follow-up build is planned that studies those tensors and holds them at 8-bit instead.
oneshot pass, sequential pipeline, two modifiers on disjoint targets: GPTQ→NVFP4 on routed experts, FP8-dynamic on attention/shared/dense-0.HuggingFaceH4/ultrachat_200k (train_sft, chat-templated) augmented with Chinese-text, code, math, and chat document slices; seed 42; activation ordering static; dampening 0.01; Hessians solved per expert (45,504 modules).nvfp4-pack-quantized in compressed-tensors): E2M1 values, group 16 on the input dimension, per-group FP8-E4M3 scales, F32 global scales shared exactly across fused groups (q/k/v; expert gate/up) — 15,168 fused pairs verified byte-equal.config.json carries one serving fix: the FP8 target list names the shared-expert projections under the flattened module path vLLM resolves at load time.Apache-2.0. Tencent's LICENSE ships verbatim in this repository. The weights here are a modified version of tencent/Hy3: quantized as described above; they are not the original weights and are not affiliated with or endorsed by Tencent. "Tencent" and "Hy3" are used descriptively to identify the source model.
<sub>Quantization, forensics, and documentation by Auroter, with AI assistance (Claude); all measurements taken and verified on the hardware named above.</sub>