Hy3-autoround-RTN-4bit-tp8-ep

Tencent Hy3 AutoRound RTN W4A16 EP (layer 0 + MTP in fp16, expert gs=128)

This repository contains a 4-bit RTN-quantized version of the Tencent Hy3 Mixture-of-Experts model, tuned for max fp8 context window on 8x3090.

Hy3 is a 295B-parameter MoE model with 21B active parameters, 192 experts (top-8 routed), and a 256K context window. Quantizing it to 4-bit with expert group_size=128 reduces the weight footprint, enabling a 174K context window with fp8 kv cache on a single 8x24 GB node.

Quantization Details

Quantized with AutoRound (0.14.0) in model-free RTN mode (calibration-free, iters=0).

auto_round /home/ubuntu/models/Hy3-tencent \
    --model_free \
    --scheme W4A16 \
    --group_size 128 \
    --layer_config "{ '.*model.layers.0.':{'bits':16,'data_type':'float'}, '.*model.layers.80.':{'bits':16,'data_type':'float'}, '.*shared_mlp':{'group_size':64} }" \
    --output_dir /home/ubuntu/models/autoround/tencent-Hy3-RTN-tp8-EP

Layer 0 is kept in full precision as the single dense FFN layer that processes every token. Layer 80 is the Multi-Token Prediction (MTP) draft layer — kept in fp16 for optimal speculative-decoding acceptance. When MTP is not enabled, layer 80 does not occupy VRAM.

⚠️ Required vLLM Branch

This model requires a patched vLLM build. Stock vLLM cannot load it due to weight-name remapping and shared-expert prefix issues:

➡️ avtc/vllm @ fix/load-quantized-hy3

Build from source:

git clone -b fix/load-quantized-hy3 https://github.com/avtc/vllm.git
cd vllm
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
TORCH_CUDA_ARCH_LIST="8.6" uv pip install --editable . --torch-backend=auto

Set TORCH_CUDA_ARCH_LIST to your GPU's compute capability to skip unused architectures and speed up the build (e.g. 8.6 for RTX 3090, 8.9 for RTX 4090, 9.0 for H100).

Verified Hardware & Performance

Verified to run with Expert Parallel on 8x NVIDIA RTX 3090 (24 GB each) GPUs.

Quick Start

export VLLM_SLEEP_WHEN_IDLE=1
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3
export RAY_memory_monitor_refresh_ms=0
export NCCL_CUMEM_ENABLE=0
export VLLM_ENABLE_CUDAGRAPH_GC=1

export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export OMP_NUM_THREADS=1

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve /path/to/tencent-Hy3-RTN-tp8-EP \
  -tp 8 \
  --enable-expert-parallel \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --port 8000 \
  --host 0.0.0.0 \
  --uvicorn-log-level info \
  --trust-remote-code \
  --max-num-seqs 4 \
  --seed 1234 \
  --max-model-len 174000 \
  --skip-mm-profiling \
  -O3 \
  --no-use-tqdm-on-load \
  --default-chat-template-kwargs '{"interleaved_thinking": true}' \
  --performance-mode balanced \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8_e4m3 \
  --gpu-memory-utilization 0.945

With MTP speculative decoding

MTP is optional and does not occupy VRAM when not enabled. Add these flags to enable Multi-Token Prediction (uses layer 80 fp16 weights and left almost no space for kv cache on 8x3090 ~10K size):

  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2

Then call the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    # reasoning_effort: "no_think" (default, direct response),
    #                   "low", "high" (deep chain-of-thought)
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

Recommended sampling parameters: temperature=0.9, top_p=1.0.

Reasoning mode: Set reasoning_effort to "high" for complex tasks (math, coding, reasoning) or "no_think" for direct responses.

Original Model

For full details on the architecture, benchmarks, and capabilities of the base model, see the original model card: ➡️ tencent/Hy3

Acknowledgments

• Tencent Hunyuan Team for the original Hy3 model.
• AutoRound team for the quantization toolkit.
• vLLM team for the inference engine.

License

This model inherits the Apache License 2.0 of the base model. See the original license.