m3-rec7-encoder

rec7 — an audio encoder for MiniMax-Music3

MiniMax-Music3 ships a language model that emits music codes and hidden states, a depth decoder, a condition encoder and a flow-matching renderer — but not the audio→codes quantizer. Nothing in the release lets you put a real recording into the model.

rec7 is a substitute for that missing piece. It reads a recording and produces, per 40 ms frame, the 4096-d hidden state the language model would have produced for that frame. From those states everything else follows deterministically using the released components: the 8 code streams, and the full condition the renderer expects.

What that enables, today:

• Covers. Real recording in → M3 renders it back. Timbre, guitar articulation and performance come through; the language model is skipped entirely. On acoustic singer-songwriter material the result is close to indistinguishable from the pipeline's own output quality.
• Supervision signal. Hidden states read from real audio are the one thing the released M3 never had. They are usable as training targets for anything that consumes those states.
• Reading codes off real audio (with an important caveat below).

Quick start

pip install -r requirements.txt
# weights/trunk.pt and weights/head.pt come with this repo
# you also need the MiniMax-Music3 checkpoint directory (not included)

# encode: states + codes for a recording
python rec7_encode.py song.mp3 --m3 /path/to/minimax_music3

# cover: render the recording back through M3
python rec7_render.py song.mp3 --m3 /path/to/minimax_music3 \
    --lyrics lyrics.txt --seconds 60 --out cover.flac

dav_loader.py (a thin loader for the Flow-VAE in dav.pth) ships alongside. The 8B language model is never loaded — the two 16k-row tables it needs are sliced straight out of the safetensors.

Memory: encoding needs ~3 GB. Covers need the full renderer (~10 GB in bf16).

Using the outputs

rec7_encode.py writes a .pt with h [T, 4096] (float16), codes [T, 8] (int16: c0 is 16384-way, c1–c7 are 1024-way), and optionally cond [T, 32768] — the renderer condition — and z, the Flow-VAE latents. 25 frames per second.

To drive the renderer yourself, rec7_model.states_to_streams() turns states into the condition, and rec7_render.render_with_condition() injects it into the pipeline (it patches the semantic generation step to hand over the states instead of running the LM).

If you build on the states, the frame→latent geometry matters: rec7_model.frame_latent_starts() is the exact piecewise mapping the renderer uses (200-frame windows on a 100-frame hop, stitched at a 345-latent hop). A constant ratio is wrong by up to a frame and it shows.

How it was made

• Architecture: conv stem + dilated residual blocks on the Flow-VAE latents (86 Hz), exact-span mean-pool to 25 Hz, 8-layer transformer (d=1088), a 2-layer head to 4096. ~170M parameters.
• Bootstrapped by self-distillation: ~56k tracks generated by M3 itself, storing codes, latents and audio, with the LM's true hidden states dumped by teacher-forcing.
• Then trained on non-synthetic audio (~22k files) with a reconstruction objective: states → released depth chain → renderer denoising step, judged against the recording's own latents. Anchored throughout by supervised code prediction on the synthetic pool so the code head doesn't drift.

Measured performance

The Flow-VAE round-trip itself preserves ~64% of 5–12 kHz energy on the same material, so the encoder slightly overshoots the high band and adds some grain — renders read a touch bright and hyped rather than dull. On sparse acoustic recordings this is much less audible than the numbers suggest.

Limitations — read these

Its codes are not plausible language-model sequences. Teacher-forced through the base LM, rec7's code sequences score ~6.2 nats per frame, against ~2.1 for the model's own samples and ~3.2–3.5 for an encoder trained to regress the LM's states directly. The states render beautifully — the renderer doesn't care about sequence plausibility — but do not use rec7's codes as training targets for the language model. Two fine-tunes trained that way collapsed into droning above modest adapter strength. Use the states; get codes elsewhere.

Timbre does not transfer through code-level training. Fine-tuning the LM (LoRA or full) on codes teaches composition, phrasing and articulation and leaves the voice generic. The voice lives in the hidden states, which code-level losses never touch.

Trained on 44.1 kHz stereo, 40 ms frames. Mono is duplicated. Very short inputs (< ~6 s) are rejected.

Training

See rec7_train.py for training script.

What we found the states are good for

The LM's own hidden states and rec7's states for the same song are related by an almost linear transform — a fitted 4096×4096 map plus offset lifts held-out agreement from 0.28 to 0.98, and a constant offset alone gets 0.86. Rendering the LM's teacher-forced states through that map produces the recording's singer, where the raw states produce a generic one. That is a route to voice identity without touching the language model; details and tooling to follow.

Files

weights/trunk.pt   the encoder (EMA weights)
weights/head.pt    the state head
rec7_model.py      architecture, geometry, loading, inference helpers
rec7_encode.py     audio -> states/codes
rec7_render.py     audio -> M3 cover
dav_loader.py      Flow-VAE loader