open-rvq-encoder-minimax-music3

Open RVQ Encoders for MiniMax Music 3

What this release does

MiniMax Music 3 normally creates its internal music codes from text and lyrics. The original audio-to-code encoder was not released. These models add an approximate path from a reference recording back to those codes.

The useful result is 0.8748 mean condition-replay cosine on 130 held-out generated tracks. In plain English: after a reference track is reduced to eight integers per 25 Hz frame and replayed through the official language-model path, v4 recovers most of the conditioning signal that the diffusion model would have received from the original codes.

This is not 87% exact-token accuracy. It is not an audio-quality score. The RVQ code space is redundant: different code tuples can lead to nearly the same downstream condition. Exact token labels therefore understate functional accuracy.

The project started with a 41M-parameter, single-GPU community proof at 0.6633 replay cosine. The SimpleTuner experiments moved through width and music-feature alignment, then reached 0.8748 by making the acoustic codebooks autoregressive across RVQ depth.

Use v4 unless reproducing an experiment.

The released ComfyUI and Diffusers adapters support one reference-generation method. Every fifth generated semantic c0 token is restricted to the encoder's top-5 candidates by default. MiniMax chooses the token and generates all acoustic codebooks. The interval is configurable from 1 through 10.

Related MiniMax Music 3 work

• SimpleTuner MiniMax Music 3 quickstart
• Fiona Crapple LM LoRA
• RVQ reference-audio integration
• Latent replanner (for synthetic audio) and experiment log
• Latent refiner (for real audio)

Files

Each weight file has a same-named .json configuration file in encoders/.

* The v1 replay evaluation used the final checkpoint. This collection ships the v1 card-recommended step-17,500 checkpoint. V2-v4 replay scores and shipped checkpoints are final checkpoints.

Metric: per-track mean cosine between replayed and stored condition embeddings. Evaluation set: 130 exact-alignment records from the held-out split. True sampled codes score 0.9999 as the pipeline control. Release provenance is in experiment-summary.json. Full summary statistics and quantiles are in evaluation/condition-replay-aggregate.json.

Experiment arc

Community proof

Serveurperso trained a 41M encoder on one GPU and demonstrated that exact token agreement was not required. Their checkpoint reached 0.6633 condition-replay cosine. Their implementation independently identified the 345-latent stitched hop, the extra warm-up code row, and the final partial-chunk interpolation rule. The SimpleTuner dataset and trainer work independently derived the same alignment contract.

That result changed the question from "can this work?" to "which errors matter downstream?"

v1: working baseline

• 41M parameters.
• DAV latents as input.
• One semantic head and seven independent acoustic heads.
• Teacher top-50 KL plus hard cross-entropy.
• μP initialization and μTransfer shape metadata.
• DDP training on the reverse-distillation corpus.
• 0.7624 replay cosine at the evaluated final checkpoint.

Detailed card: SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1.

v2: width

• Width increased from 512 to 1,088.
• Parameters increased from 41M to 155M.
• Polynomial decay with warm-up.
• Replay improved from 0.7624 to 0.7698.

Width helped token metrics. It barely moved the downstream replay metric. Capacity was not the main limit.

Detailed card: SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2.

v3: MERT alignment

• Same exported architecture as v2.
• Frozen MERT features used only as a training target.
• Mid-layer cosine alignment loss annealed to zero.
• No MERT weights or projection are present in the exported encoder.
• Replay improved from 0.7698 to 0.7703.

The measured gain was too small to justify an inference dependency or a larger MERT experiment on this corpus.

Detailed card: SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3.

v4: autoregressive depth

• Shared encoder kept at v2 width.
• The seven acoustic heads were replaced by a causal decoder.
• Codebook k is conditioned on the semantic code and acoustic codebooks < k.
• Parameters increased to 169M.
• Replay improved from 0.7703 to 0.8748.

Free-running exact acoustic top-1 decreased slightly. Teacher-forced acoustic top-1 increased to 18.42%. Replay improved by 0.1045. This is the central result: code-tuple compatibility matters more than matching one sampled token tuple.

Detailed card: SimpleTuner/open-rvq-encoder-minimax-music3-169m-v4.

ComfyUI

Requirements:

• current ComfyUI with native MiniMax Music 3 support;
• official ComfyUI MiniMax diffusion model, text encoder, and DAV decoder;
• original dav.pth, because the Comfy DAV file is decoder-only;
• this repository cloned with Git LFS.

git lfs install
git clone https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3

cd ComfyUI
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt huggingface_hub

.venv/bin/hf download Comfy-Org/MiniMax-Music-3 \
  diffusion_models/minimax_music3_dit_fp16.safetensors \
  text_encoders/minimax_music3_text_encoder_pruned_int8_convrot.safetensors \
  vae/minimax_music3_dav.safetensors \
  --local-dir models

.venv/bin/hf download MiniMaxAI/MiniMax-Music3 dav.pth \
  --local-dir models/vae

ln -s /path/to/open-rvq-encoder-minimax-music3/comfyui_open_rvq \
  custom_nodes/comfyui_open_rvq

Restart ComfyUI. Upload a reference audio file and select v4 in MiniMax Music3 RVQ Reference Encoder Loader.

• comfyui_workflow_example.json constrains every fifth generated semantic c0 token to the encoder's top-5 candidates.
• reference_interval=1 constrains every frame.
• reference_interval=5 is the tested default.
• reference_interval=10 constrains every tenth frame and gives the language model more freedom.

The seven acoustic codebooks are generated by MiniMax. They are not copied from the reference. Describe the target arrangement in caption. Provide the desired sectioned lyrics. Prompt adherence and audio quality vary; this is not a general audio-to-audio conversion system.

The node package reads the encoder files directly from this clone. They can instead be placed in:

ComfyUI/models/minimax_music3_rvq_encoders/

The interval-5 graph was verified with a 30-second reference, five Euler steps, the pruned int8 text encoder, the fp16 diffusion model, and the v4 encoder. It produced a 29.99-second stereo 44.1 kHz FLAC. Use 30 diffusion steps for normal output.

Diffusers

The MiniMax Music 3 Diffusers integration is currently installed from its integration commit:

python3 -m venv .venv
source .venv/bin/activate
pip install \
  git+https://github.com/huggingface/diffusers@dafe3733fcfdbf3c48915fe77be3aef65b5d6a2d \
  transformers accelerate safetensors huggingface_hub soundfile torchaudio

Download minimax_music3_reference_adapter.py into the project, then:

import soundfile as sf
import torch
from diffusers import ModularPipeline

from minimax_music3_reference_adapter import (
    MiniMaxMusic3ReferenceAdapter,
    install_diffusers_reference_adapter,
)

install_diffusers_reference_adapter()

pipe = ModularPipeline.from_pretrained(
    "MiniMaxAI/MiniMax-Music3",
    torch_dtype=torch.bfloat16,
).to("cuda")
adapter = MiniMaxMusic3ReferenceAdapter.from_pretrained()

audio, sample_rate = sf.read("reference.flac", always_2d=True)
waveform = torch.from_numpy(audio.T).float()
generator = torch.Generator(device="cpu").manual_seed(42)

frame_hiddens, predicted_codes = adapter.encode_reference(
    pipe,
    waveform,
    sample_rate,
    prompt="rock",
    lyrics="[instrumental]",
    generator=generator,
    device="cuda",
    reference_interval=5,
)

result = pipe(
    frame_hiddens=frame_hiddens,
    generator=torch.Generator(device="cpu").manual_seed(42),
    num_inference_steps=30,
    output_type="pt",
)

reference_interval accepts integers from 1 through 10. The patch only adds a precomputed-frame_hiddens bypass to the modular pipeline. It does not replace MiniMax model code.

Limits

• These are approximate encoders, not the original MiniMax encoder.
• No original encoder weights or source were used.
• Training data is synthetic MiniMax Music 3 output, not MiniMax's training set.
• Real-audio generalization is not established.
• Context is 128 frames, or 5.12 seconds. There is no cross-window encoder state.
• Reference generation still needs the official MiniMax language model and RVQ depth decoder.
• The released integration uses the RVQ encoder's top-5 semantic candidates. Other candidate counts are not exposed.

Credits

• bghira: reverse-distillation dataset, independent alignment derivation, SimpleTuner trainer, compute, v1-v4 runs, and replay experiments.
• marduk191: WAV/code samples, early mel-encoder proof, corpus-scale observations, and encoder experiments.
• scragnog: SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
• Serveurperso: independent 41M encoder, single-GPU proof, replay evaluation, and independent stitched-timeline findings.
• dernet: RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.

License

minimax_music3_reference_adapter.py and comfyui_open_rvq/ are Apache-2.0 licensed. See LICENSE-CODE.

The encoder weights were trained on outputs from MiniMax Music 3. Use of the weights, official model components, and generated output remains subject to the applicable MiniMax Music 3 terms. The MERT license also applies to the v3 training provenance; no MERT weights are distributed here.