uvr-mdxnet-kara2-onnx

UVR-MDX-NET KARA 2 — lead/backing vocal separation (ONNX, WebGPU)

Musetric no longer runs this graph: since musetric/musetric#1157 its separation model, musetric/mel-band-roformer-duality-kara2-student-onnx, gives the lead and the backing in one pass, with a split distilled from KARA 2. This repository stays for the builds that pin it.

ONNX graph of the UVR-MDX-NET KARA 2 karaoke model from Ultimate Vocal Remover, packaged for the musetric packages/ai runtime (onnxruntime-web on WebGPU).

The graph is the network only: it takes a complex spectrogram chunk and returns the complex spectrogram of everything except the lead vocal. The weights and operators are those of the UVR release file; the time axis of the U-Net is carried as bands in the batch axis, so that the WebGPU execution provider computes it correctly on Adreno 6xx GPUs too (see the known defect below). STFT, chunking, overlap-add and the lead/backing split run in the host. This is not a drop-in audio -> stems model.

vocal stem, 44100 Hz stereo, peak-normalized
  -> STFT (n_fft 5120, hop 1024, periodic Hann, reflect-padded center frames)
  -> bins 0..2047, bins 0..2 zeroed -> input [1, 4, 2048, 256] per chunk
  -> kara2.onnx                     -> output [1, 4, 2048, 256] per chunk
  -> bins 2048..2560 zeroed, iSTFT, overlap-add across chunks -> separated
  -> backing = separated · peak
  -> lead    = (vocal − 1.065 · separated) · peak

Intended uses & limitations

Intended:

• Splitting an isolated vocal stem into lead and backing vocals, as the second separation stage of an audio pipeline (after vocals/instrumental).
• Client/edge inference via WebGPU through onnxruntime-web, on desktop and on mobile GPUs alike. The wasm EP gives the same result.

Out of scope:

• Use on full mixes: the model expects a vocal stem.
• Standalone use without a host that computes the STFT input and inverts the output (see the host code below).
• Use in other training frameworks — this is an inference-only graph.

Limitations:

• The host contract is exact. 44100 Hz stereo; n_fft 5120, hop 1024, periodic Hann window 0.5 − 0.5·cos(2πn/N), frames centered with reflect padding of n_fft/2; the first 2048 of 2561 bins, with bins 0–2 zeroed; channels ordered [left real, left imaginary, right real, right imaginary]. A different window, bin count or channel order gives wrong output without an error.
• One static chunk: batch 1 and 256 frames (261,120 samples). The source graph declared a dynamic batch; this one does not, since its batch axis carries the time bands inside.
• The lead stem is a residual with UVR's compensation factor 1.065, not a second model output.
• Long GPU stretches on phones. Each full-resolution convolution is one dispatch of 10.87 GMAC; on Adreno 660 another GPU client waits up to about a second for its turn while a chunk runs (see Validation).
• Training-data provenance of the upstream weights is not documented upstream.

Known defect: onnxruntime WebGPU on Adreno 6xx, and the time bands (2026-10-04)

onnxruntime-web runs Conv, ConvTranspose and BatchNormalization on WebGPU in NHWC and inserts a layout Transpose around each of them. For a batch of one, that Transpose reduces to a single [C, H·W] matrix swap and takes a tiled kernel that loses a quarter of its writes on Adreno 6xx; the source graph then comes out silently wrong there (2,094,338 of 2,097,152 elements off by more than 1e-2 on an Adreno 660, and different from run to run). The operators themselves compute correctly. Probed one by one on an Adreno 660 with this model's weights:

This revision keeps every operator of the source graph and holds each tensor of the U-Net body, [1, C, 256, F] in the source, as [4, C, 64, F]: four bands of 64 time frames, one per batch item, so every layout transpose swaps a batch of matrices. Only a 3×3 convolution mixes neighbouring frames; it takes its halo rows from the neighbouring bands (zeros at the ends of the plane) and runs with no padding along time. The 2×2 strided and transposed convolutions pair frames inside a band; the frequency-axis MatMuls, BatchNormalization, Relu, Add and Mul act per frame. The input 1×1 convolution runs on the bands; the output 1×1 convolution runs as a MatMul followed by a rank-3 Reshape before the bands are joined back into time, so that no layout transpose lands on a batch of one. Rebuild: fold_time_bands.py --bands 4 in musetric-toolkit scripts/onnx/kara2/.

Revisions 7d49dab and 8f6d5fe shipped a static rewrite instead, with every convolution unrolled into MatMuls (8f6d5fe cut into row bands); it was correct everywhere and 1.1–2.1 times as slow.

How to use

The session runs one chunk; the host supplies input and consumes output.

import * as ort from 'onnxruntime-web/webgpu';

const session = await ort.InferenceSession.create('kara2.onnx', {
  executionProviders: [{ name: 'webgpu', storageBufferCacheMode: 'simple' }],
});

// spectrum: Float32Array of shape [1, 4, 2048, 256], produced host-side from one
// 261,120-sample stereo chunk of the vocal stem (see the contract above).
const input = new ort.Tensor('float32', spectrum, [1, 4, 2048, 256]);
const { output } = await session.run({ input });
// output: float32 [1, 4, 2048, 256] -> iSTFT and overlap-add host-side.

See the musetric host code at 33d010c8, the last to run this graph (packages/ai/src/runtime/leadBacking/, packages/server/src/analysis/voices.rs), for the full STFT, chunk plan and lead/backing split.

Files

Signature — float32 weights, opset ai.onnx 13, IR 6:

Validation

Against the source graph. On the onnxruntime CPU provider the output equals the source graph's exactly on the deterministic input. On WebGPU it equals the source graph run on the same device on desktop NVIDIA, Adreno 750 and Apple M3 (iPad Air), and it is the same on Adreno 750 and Adreno 660.

Parity. The musetric parity gate (torch reference, ONNX on the CPU, wasm, WebGPU and the product's runtime, on two cases of an openly licensed song) passes on desktop NVIDIA, Adreno 750, Adreno 660 and Apple M3; the product's chunk output lies 111.7–117.3 dB from the original model.

Speed and responsiveness, against the previous revision 8f6d5fe, as musetric runs a chunk (paced one submission at a time), three loads of each revision alternated. The foreign job wait is how long a one-dispatch job submitted every 100 ms on another GPU device waits for its turn:

Re-run the parity gate on the exact published bytes before relying on it.

Source & lineage

Code license and weight license are separate; ONNX conversion does not change the weight license. Documented only as far as it is verifiable.

• Architecture: MDX-Net (TFC-TDF U-Net) over a complex spectrogram: a 1×1 input convolution to 48 channels; at each of six resolutions (48 → 288 channels, 2048 → 64 frequency bins) three 3×3 conv–ReLU layers and a residual dense block that squeezes the frequency axis 8× with batch norm and ReLU; 2×2 strided convolutions down, 2×2 transposed convolutions with batch norm and ReLU up, skip connections merged by multiplication; a 1×1 output convolution. 13.19 M parameters. Original MDX-Net code: kuielab/mdx-net.
• Weights: Ultimate Vocal Remover (Anjok07, aufr33), MIT.
• Source file: UVR_MDXNET_KARA_2.onnx from the UVR model release TRvlvr/model_repo all_public_uvr_models — 52,786,726 B, sha256 bf32e15105a09c0f7dddd2b67346146334d6f3ecb399ed7638eba2ab07cbf5f4. A release asset is not content-addressed, so the sha256 is the pin.
• Export tooling: scripts/onnx/kara2 in musetric-toolkit.
• Host runtime: packages/ai and packages/server in musetric.

This graph preserves the upstream MIT license; we do not claim authorship of the original weights.

License

MIT, inherited from the upstream weights.