Hy4-preview-NVFP4-W4A16

Hy4-preview — NVFP4 W4A16 (experts-only)

Independent NVFP4 quantization of tencent/Hy4-preview (780B MoE + MTP) by Oxmiq Labs, for single-node 8× ~96 GB Blackwell (SM120) serving via the official vllm/vllm-openai:hy4-preview image.

Not affiliated with, or endorsed by, Tencent. Produced independently from the publicly released base model under its Apache-2.0 license.

Only the routed experts (744.10B params, ~95.4% of the 780B total) are quantized to NVFP4. Layer 0 (the dense FFN), router gates, DSA indexer, MLA attention, shared experts, embeddings, fp32 lm_head, iHC parameters, and the MTP head stay in their original precision.

On the "8-bit precision" badge: Hugging Face infers precision from the safetensors dtype. compressed-tensors packs two 4-bit NVFP4 values into each uint8 byte, so the sidebar shows U8 / "8-bit precision". The actual weight format is 4-bit e2m1, with one e4m3 scale per 16-element group and one fp32 global scale per tensor — about 4.5 bits per weight including scales.

Hardware requirements

Reference target for the recipe below:

This build runs as W4A16 via Marlin: 4-bit weights are unpacked to 16-bit inside the matmul kernel, and all math runs at 16-bit. It doesn't use SM120's native FP4 tensor cores, so it doesn't depend on FP4-activation kernel support. Other GPU counts or memory sizes need a different parallelism and --max-model-len plan; the flags below are the 8×96 GB configuration we run, not the only possible one.

Method

Weights-only RTN NVFP4 (fp4 e2m1 + e4m3 group scales per 16 elements + fp32 global scale), produced with llmcompressor model_free_ptq directly on safetensors shards (no transformers model class; experts are fused nn.Parameter tensors).

• On-disk: compressed-tensors (nvfp4-pack-quantized / W4A16).
• 59,136 expert tensors quantized; sensitive tensors untouched.
• MTP head preserved and re-fused byte-identical to source for vLLM’s MTP layout.
• Pinned toolchain: llmcompressor 0.13.0 · compressed-tensors 0.18.0 · torch 2.13 · Python 3.12.

Artifact inventory

• model-00001-of-00131.safetensors … model-00131-of-00131.safetensors — 130 weight shards. There is no model-00042-of-00131.safetensors; that number is skipped, so the highest index is 00131 while the file count is 130. Nothing is missing: model.safetensors.index.json maps every tensor across exactly those 130 files.

• model.safetensors.index.json — weight index.

• hy4-manifest-20260910.md5 — md5 checksums covering all 130 shards. Verify a download with:

  md5sum -c --ignore-missing hy4-manifest-20260910.md5
  

  --ignore-missing is required. The manifest was generated against a tree that also held upstream files not shipped here (finetune/, assets/, README_CN.md), and without the flag those 30 entries report as failures on an otherwise intact download. Files added after the manifest — this README, benchmarks/, the serve scripts — are not covered by it.

• serve_entrypoint.sh — applies both required SM120 hot-patches (#54563 inline, #54568 via patch_54568.py), then execs the validated serve argv.

• patch_54568.py — the #54568 attention-sinks patch, invoked by serve_entrypoint.sh.

• benchmarks/ — raw result JSON for the published throughput tables, with field notes.

• Tokenizer / chat template / config.json / generation_config.json / LICENSE (Apache-2.0)

Validation

• Selection audit vs real index: 59,136 experts selected; zero sensitive tensors touched
• MTP re-fuse: byte-identical to source
• Post-run format checks: 8/8 passed (schema, census, dtypes, index, size ratio ~0.314)
• Weight fidelity (Tier-1): median rel-err 0.0951, p99 0.0952 (n=66 stratified expert samples; cosine ≥ 0.9955)
• End-to-end load on SM120 with official vllm/vllm-openai:hy4-preview (after required SM120 patches — see Serving)
• Load sanity: 4/4 basic generation checks correct on the NVFP4 serve path

Evaluation results

Measured on the NVFP4 serve path used during validation. Not a claim of parity with official BF16/FP8 leaderboards. Protocols matter; each row states its own.

Pending: CoT/think-mode MMLU-Pro. Optional follow-ups not claimed here: full RULER, IFEval, HumanEval, GPQA.

No same-harness BF16/FP8 baseline is reported yet, so the figures above are absolute scores on this serve path rather than measured deltas against the unquantized model.

Serve throughput (hy4-preview-nvfp4, API)

Measured 2026-09-24 against a live hy4-preview-nvfp4 OpenAI-compatible endpoint (same served-model name as this card’s entrypoint). Workload 1024 in / 1024 out, 100 prompts at each concurrency. Not a same-box claim for every 8× host; use as a reference point next to the Quick start recipe.

Measured with OxCapsule's benchmarking feature (cap benchmark), built on the open-source InferenceMAX framework and vLLM's benchmark_serving client (Apache-2.0).

Measurement path: these figures were taken through an OpenAI-compatible gateway sitting in front of the engine, not against a bare vllm serve as set up by the Quick start recipe below. The same workload measured directly against the engine, with the same flags, returns throughput within 2% and identical per-token latency, so the recipe reproduces this table. The one column the gateway moves is TTFT, and it moves it upward: 376 ms direct against 436 ms here at concurrency 16. A bare deployment should be slightly faster to first token, not slower.

Aggregate output throughput scales monotonically with concurrency. Per-stream decode falls from ≈29 to ≈19 tok/s as batching increases, which is the expected trade.

The Completed column counts every request that finished, so it accounts for incomplete and timed-out requests as well as hard failures.

Raw result JSON for each run above is in benchmarks/, with field notes and caveats in benchmarks/README.md.

Cross-check against vllm bench serve

The table above is measured with OxCapsule. As an independent check, the same endpoint and workload were measured with vLLM's own vllm bench serve at concurrency 16.

Same OpenAI-compatible endpoint, 1024 in / 1024 out, 100 prompts, concurrency 16, greedy sampling, 3 warmup requests. Both tools measured through the same gateway endpoint, not against the engine directly, so these figures are comparable to each other and to the table above rather than to a bare vllm serve deployment. Raw output in benchmarks/vllm_bench_serve_gateway_c16.json.

Inter-token latency and end-to-end latency agree to within 0.3% and 0.1%. TTFT differs by 75% between the two tools on this path. Note that TTFT is highly sensitive to warmup: the same tool against the same endpoint returned 6,752 ms with no warmup requests and 376 ms with three, so any TTFT figure should state its warmup count.

Throughput is deliberately not compared. When measured through a proxy that returns a non-empty choices array in the same chunk as usage, vllm bench serve skips its usage branch and falls back to tokenizing the assistant content it accumulated. Because this model is served with --reasoning-parser hy_v4, the reasoning tokens sit in a separate field and go uncounted: it recorded 28.9 output tokens per request for requests that each ran about 54 seconds at roughly 51 ms per token. Measured against the engine directly the same tool counts correctly. Its latency measurements are unaffected on either path.

Serving (the recipe we run)

Stack: vllm/vllm-openai:hy4-preview · --quantization compressed-tensors · TP=8, PP=1 · Marlin W4A16 (activations stay 16-bit).

Locked flags (silent failure if wrong)

These do not always fail loudly — wrong values produce hangs, OOMs, extreme slowdowns, or degenerate / looping output:

Required SM120 hot-patches

Until upstream merges land in the image you pull, the stock hy4-preview image still needs:

1. vLLM #54563 — KV tile-scale format for hy_v4 (#54434)
2. vLLM #54568 — forward attention sinks through sparse MLA

Without both, the engine can look healthy and still emit garbage. This repo ships serve_entrypoint.sh (applies #54563 + runs patch_54568.py) so you do not have to invent the patch steps.

Quick start (recommended)

Assumes Docker with NVIDIA Container Toolkit and an 8× ~96 GB SM120 host (see Hardware requirements). The Hub repo is public — hf download works without a token. Optional: set HF_TOKEN or run hf auth login for higher rate limits.

1. Install the Hugging Face CLI (if needed). huggingface-cli / hf-cli are deprecated — use hf:

curl -LsSf https://hf.co/cli/install.sh | bash

1. Download the checkpoint and scripts:

hf download Oxmiq/Hy4-preview-NVFP4-W4A16 \
  --local-dir ./Hy4-preview-NVFP4-W4A16
# optional: --token "$HF_TOKEN"

1. Make the entrypoint executable:

chmod +x ./Hy4-preview-NVFP4-W4A16/serve_entrypoint.sh

1. Pull the official serve image:

docker pull vllm/vllm-openai:hy4-preview@sha256:dc3f5fbe373b74141605a1e7843ae305faa648e584cac73a201395cc259522f6

Pinned to the image digest this card was validated against; the floating hy4-preview tag can move.

1. Serve via the repo entrypoint (applies #54563 + #54568, then the locked argv):

docker run --gpus all --ipc=host -p 8000:8000 \
  -e NCCL_P2P_DISABLE=1 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e TORCH_CUDA_ARCH_LIST=12.0 \
  -e VLLM_ENABLE_HPC_OPS=1 \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -v "$PWD/Hy4-preview-NVFP4-W4A16:/model" \
  -v "$PWD/Hy4-preview-NVFP4-W4A16:/work" \
  --entrypoint bash \
  vllm/vllm-openai:hy4-preview@sha256:dc3f5fbe373b74141605a1e7843ae305faa648e584cac73a201395cc259522f6 \
  /work/serve_entrypoint.sh

Cold first launch may spend about 15 minutes in FlashInfer JIT; a persistent FLASHINFER_WORKSPACE_BASE cache avoids repeating that. Single-stream decode is in the about 30 tok/s class on the measured hy4-preview-nvfp4 path (see Serve throughput); workload-dependent.

1. Smoke test (this is a reasoning model — use enough tokens):

curl -s http://127.0.0.1:8000/v1/models | head
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "hy4-preview-nvfp4",
    "messages": [{"role":"user","content":"Say hello in one short sentence."}],
    "max_tokens": 200,
    "temperature": 0
  }'

Pass if HTTP 200, the reply is coherent (reasoning then answer is fine), and there are no single-token / phrase loops. Failures that usually mean a bad launch: OOM, NCCL hang, or looping garbage (missing patches, MTP left on, or wrong GPU shape).

serve_entrypoint.sh applies the patches, then runs:

vllm serve /model
  --served-model-name hy4-preview-nvfp4
  --quantization compressed-tensors
  --tensor-parallel-size 8
  --pipeline-parallel-size 1
  --distributed-executor-backend mp
  --hf-overrides '{"num_nextn_predict_layers":0}'
  --kv-cache-dtype fp8_ds_mla
  --gpu-memory-utilization 0.92
  --max-num-seqs 16
  --max-num-batched-tokens 8192
  --enable-chunked-prefill
  --enable-prefix-caching
  --tool-call-parser hy_v4
  --reasoning-parser hy_v4
  --enable-auto-tool-choice
  --trust-remote-code
  --port 8000
  --attention-backend FLASHINFER_MLA_SPARSE_SM120
  --max-model-len 262144

Advanced: bare docker run (same flags; still need patches)

Prefer Quick start above. Only use this if you apply #54563/#54568 inside the container yourself before vllm serve (same flags as the entrypoint; skipping patches can look healthy and still emit garbage):

docker run --gpus all --ipc=host -p 8000:8000 \
  -e NCCL_P2P_DISABLE=1 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e TORCH_CUDA_ARCH_LIST=12.0 \
  -e VLLM_ENABLE_HPC_OPS=1 \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -v /path/to/Hy4-preview-NVFP4-W4A16:/model \
  vllm/vllm-openai:hy4-preview@sha256:dc3f5fbe373b74141605a1e7843ae305faa648e584cac73a201395cc259522f6 \
  /model \
  --served-model-name hy4-preview-nvfp4 \
  --quantization compressed-tensors \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 1 \
  --distributed-executor-backend mp \
  --hf-overrides '{"num_nextn_predict_layers":0}' \
  --kv-cache-dtype fp8_ds_mla \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --enable-auto-tool-choice \
  --trust-remote-code \
  --attention-backend FLASHINFER_MLA_SPARSE_SM120 \
  --max-model-len 262144

Compatibility

• vLLM: official hy4-preview image (hy_v4 via vLLM #54160+)
• Hardware: SM120 Blackwell ~96 GB × 8 is the configuration we ran; FP4-activation hardware not required for this W4A16 build
• Not a drop-in for stock unpatched SM120 sparse-MLA without the KV/sinks fixes above
• SGLang: not treated as an escape hatch for this card (hy_v4 support timing differs)

Related checkpoints

Several Hy4 low-bit builds exist, targeting different runtimes and hardware:

• 0xTank/Hy4-preview-NVFP4-W4A16-MTP — NVFP4 via ModelOpt, custom Spark runtime
• mlx-community/Hy4-preview-4bit — MLX / Apple Silicon, ~433 GB
• inferencerlabs/Hy4-preview-MLX-Q4i — MLX / Inferencer app, Apple Silicon, ~456 GB
• AngelSlim/Hy4-preview-GGUF — GGUF / llama.cpp
• qtum/Hy4-preview-GGUF — GGUF / llama.cpp

This checkpoint is the compressed-tensors NVFP4 W4A16 build aimed at 8× ~96 GB Blackwell (SM120) with the official vllm/vllm-openai:hy4-preview image (~490 GB / ~457 GiB on disk).

Size differences between these builds largely reflect how much of the model each one converts. Here the conversion is limited to the routed experts; attention, shared experts, router gates, layer 0 (dense FFN), the MTP head and an fp32 lm_head stay at source precision, which accounts for most of the difference against the MLX builds.

License & attribution

Apache-2.0, same as tencent/Hy4-preview. LICENSE is included. Significant changes: routed-expert weights → NVFP4; re-shard; MTP experts re-fused to source layout. Not affiliated with or endorsed by Tencent (see the note at the top of this card).

Known limitations

• Quality vs BF16/FP8 will differ on expert-heavy tasks; read the eval table and protocols
• MMLU-Pro is not reported; a CoT/think-mode run is pending.
• Data-free RTN (no calibration set), by design
• MTP is shipped but must be disabled. The MTP head is preserved byte-identical to source, but constructing it on this W4A16 path causes hard single-token loops, so the serve configuration above passes --hf-overrides '{"num_nextn_predict_layers":0}'. Speculative decoding via MTP is therefore not available with this checkpoint as published; acceptance on W4A16 has not been measured.
• 1M context is an architecture claim of the base model; this card only reports NIAH hits through 128k on the NVFP4 path, and the serve configuration above pins --max-model-len 262144
• Shard filenames use of-00131 with unused index 42 (cosmetic; not present in the upstream BF16 shard layout — no tensors point at that shard)