YuE2-3B-OrbitQuant-W4A4

YuE2-3B · OrbitQuant W4A4

Music generation from style and lyrics (m-a-p/YuE2-3B) with packed 4-bit transformer weights and activations. Output: 48 kHz stereo.

Model

• 392 transformer projections in OrbitQuant W4A4 (224 packed modules). Embeddings, norms, auxiliary projections and the audio VAE stay in source precision; output heads are INT8 or W4A4 only in the profiles below.
• Checkpoint 3.04 GB vs 7.26 GB BF16 (−58.2%). Separate 0.53 GB VAE unchanged. No fine-tuning or distillation.

Run

hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
cd YuE2-W4A4 && python3.12 -m venv .venv && source .venv/bin/activate
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
(cd src && pip install -r requirements.txt)
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --mode fast --output out

One request per process is the default. A long-lived process serving many requests adds --warm-compile (compiled VAE decoder: 0.2 s instead of 0.7 s per request after a one-time 2 s cache load).

Requirements: Linux x86_64, Python 3.12, PyTorch 2.10.0 + CUDA 12.8; the OrbitQuant 0.9.8 wheel is bundled in src/wheels, native kernels 1.0.5 under src/runtime/kernels. Kernels: SM89 (validated on RTX 4090) and SM120 (built, not run). The first request compiles the VAE decoder (~25 s, cached afterwards).

Measurements

RTX 4090 (48-core host), identical input and seed. "Warm" is the second request of a long-lived process with --warm-compile; "process" is one src/run.py call from interpreter start to the written files (imports and model load included). Peaks are for the whole process.

exact keeps the off tokens and logits bit for bit; its waveform differs from off only at FP16 rounding in the VAE. INT8/W4A4 heads change sampling at the same seed, so compare those modes by audio seconds per wall second. Of a whole process, about 4 s are Python and library imports plus model load; the rest is generation. Wall time also depends on the host CPU: on a slow 4-core host the same exact request took 17.7 s and off 28.3 s. Raw data: evaluation/profiles/.

Files

src/: runtime, kernels and kernel sources; src/kernel-binaries.json lists the binaries per architecture, src/runtime-lock.json pins versions, SHA256SUMS covers every file. examples/: prompt file and WAV demos (BF16, W4A4, fast INT8 KV).

License: CC BY-NC 4.0 for the weights (inherited from M-A-P / YuE2); code components keep their own licenses. No commercial rights.