Hy3-REAP-48e

Hy3-REAP-48e — REAP-pruned Hunyuan Hy3 (experimental)

An expert-pruned version of tencent/Hy3, produced with REAP (Router-weighted Expert Activation Pruning). Each MoE layer's 192 routed experts are reduced to the 48 most salient, shrinking the checkpoint from 557 GB to 157 GB (BF16).

⚠️ Experimental research artifact. This is an aggressive prune (75% of experts removed) calibrated on a small, code-only set. It generates fluent text and is genuinely strong at code, but general/factual accuracy is degraded (see below). Not intended for production. A better-calibrated version is planned — this repo will be updated in place.

How it was made

The entire pipeline ran on a single NVIDIA DGX Spark (128 GB) — a model far larger than RAM — using per-layer disk streaming:

• Method: REAP, one-shot (no retraining). Each expert is scored by the mean of router_weight × activation_norm over a calibration set; the least-salient are dropped.
• Compression: compression_ratio 0.75 → 192 → 48 experts kept per MoE layer.
• Calibration: 64 samples from theblackcat102/evol-codealpaca-v1 (code).
• Streaming: the 557 GB model was built on meta and each of the 80 layers streamed from the on-disk safetensors shards (~7 GB resident at a time), for both calibration and the prune. See the How To Spark training notes for the recipe.

Quality profile (sanity check)

Fluent everywhere; quality tracks the (code-only) calibration domain:

The pattern is expected: code experts (the calibration domain) survived; experts carrying broad world-knowledge scored low on code data and were pruned. A larger, more diverse calibration set and/or a gentler prune ratio would recover accuracy.

Running it

At 157 GB BF16 this exceeds a single 128 GB device. Options:

• Two GPUs / two nodes — e.g. vllm serve <repo> --pipeline-parallel-size 2 (vLLM supports HYV3ForCausalLM). This model was validated running pipeline-parallel across two DGX Sparks.
• 4-bit quantization (~40 GB) to fit a single device (tooling for hy_v3 quantization is still maturing).

License

Inherits the base model's license (Apache-2.0). Derivative of tencent/Hy3.