Hy3-GPTQ-RTN-4bit-tp8

Tencent Hy3 GPTQ RTN 4bit tp8 (MPT layer removed)

This repository contains a 4-bit RTN-quantized version of the Tencent Hy3 Mixture-of-Experts model.

Hy3 is a 295B-parameter MoE model with 21B active parameters, 192 experts (top-8 routed), and a 256K context window. Quantizing it to 4-bit brings the weight footprint down to ~150 GB, making it possible to serve the full 256K context on a single 8x24 GB node with KVarN KV cache compression.

⚠️ Required vLLM Branch

This model requires a patched vLLM build. Stock vLLM cannot load it due to several bugs in weight-name remapping, dynamic GPTQ MoE group-size handling, and TurboQuant KV cache shaping:

➡️ avtc/vllm @ fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26

The branch includes these fixes:

• fixes to allow run AWQ/GPTQ quants of Mimo-v2.5
• chery picked commit: [Attention] Add KVarN: calibration-free variance-normalized KV-cache quantization backend from https://github.com/vllm-project/vllm/pull/46812

Build from source:

git clone -b fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26 https://github.com/avtc/vllm.git
cd vllm
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
TORCH_CUDA_ARCH_LIST="8.6" uv pip install --editable . --torch-backend=auto

Set TORCH_CUDA_ARCH_LIST to your GPU's compute capability to skip unused architectures and speed up the build (e.g. 8.6 for RTX 3090, 8.9 for RTX 4090, 9.0 for H100).

Quantization Details

Quantized with GPTQModel (7.2.0) using the RTN (Round-To-Nearest) fallback strategy.

Verified Hardware & Performance

This model is verified to run with Tensor Parallel on 8x NVIDIA RTX 3090 (24 GB each) GPUs with the full 256K (262,144) context window, using kvarn_k4v2_g128 4-bit K, 2-bit V near lossless cache compression.

• Tensor parallel: 8 (no expert parallel — expert group size 64 divides cleanly across 8 GPUs)
• KV cache: kvarn_k4v2_g128
• Max model len: 262,144
• Model weight footprint: ~150 GB
• Reasoning (<think>) and tool-calling work.

Quick Start

export VLLM_SLEEP_WHEN_IDLE=1
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3
export RAY_memory_monitor_refresh_ms=0
export NCCL_CUMEM_ENABLE=0
export VLLM_ENABLE_CUDAGRAPH_GC=1

export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export OMP_NUM_THREADS=1

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve /path/to/tencent-Hy3-RTN \
  -tp 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --port 8000 \
  --host 0.0.0.0 \
  --uvicorn-log-level info \
  --trust-remote-code \
  --max-num-seqs 4 \
  --seed 1234 \
  --max-model-len auto \
  --skip-mm-profiling \
  -O3 \
  --no-use-tqdm-on-load \
  --default-chat-template-kwargs '{"interleaved_thinking": true}' \
  --performance-mode balanced \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --kv-cache-dtype kvarn_k4v2_g128 \
  --gpu-memory-utilization 0.945

Then call the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    # reasoning_effort: "no_think" (default, direct response),
    #                   "low", "high" (deep chain-of-thought)
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

Recommended sampling parameters: temperature=0.9, top_p=1.0.

Reasoning mode: Set reasoning_effort to "high" for complex tasks (math, coding, reasoning), "low" for faster thinking or "no_think" for direct responses.

Verified Functionality

• ✅ Functional 2D HTML aquarium generation (without reasoning enabled) — working artifact.
• ✅ Reasoning mode (<think>) engages and produces output.

Original Model

For full details on the architecture, benchmarks, and capabilities of the base model, see the original model card: ➡️ tencent/Hy3

Acknowledgments

• Tencent Hunyuan Team for the original Hy3 model.
• GPTQModel team for the quantization toolkit.
• vLLM team for the inference engine.
• KVarN team for abilty to use large near-lossless KV cache

License

This model inherits the Apache License 2.0 of the base model. See the original license.

Attribution

@misc{muller2026kvarn,
      title={KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks}, 
      author={Lorenz K. Muller and Philippe Bich and Chiara Boretti and Hyun-Min Chang and Jiawei Zhuang and Lukas Cavigelli},
      year={2026},
      eprint={2606.03458},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={http://arxiv.org/abs/2606.03458}
}