Motif-Audio

<!-- TODO before public release: add paper and demo/Space links to Model Sources. -->

Motif-Audio: A General-Purpose Audio Foundation Model

Motif-Audio is a general-purpose audio foundation model. One model covers work that usually needs three: understanding sound, compressing and rebuilding it, and giving generative models a space to build in.

• Understand. It reads emotion in a voice, genre in a song, and events in everyday sound, and those abilities carry from one task to the next.
• Reconstruct. As a continuous neural codec, it compresses audio to a compact continuous representation and rebuilds the waveform at quality on par with the best neural codecs.
• Generate. It gives text-to-audio and music generation models a foundation to build on. They learn to produce its latent, and the model turns that into sound.

The design is a dual-stream autoencoder. One stream follows content, the other follows fine spectral detail, and a cross-attention fusion brings them together. That separation is what lets a single model do the work of three.

Model details

• Developed by: Motif Technologies
• Model type: dual-stream audio autoencoder
• Sample rate: 16 kHz
• Latent: continuous, width 1024
• Parameters: 726M total (605M frozen semantic encoder; 121M trained: acoustic encoder, fusion, decoder)
• License: MIT

Architecture

<p align="center"> <img src="assets/architecture.png" alt="Motif-Audio dual-stream architecture: a frozen MSM-MAE semantic stream and a lightweight Conv2d patch-embed acoustic stream are combined by cross-attention fusion into a continuous latent, which a Vocos decoder and iSTFT turn back into a 16 kHz waveform" width="900"> </p>

Motif-Audio runs a raw 16 kHz waveform through two parallel encoders, a cross-attention fusion, a continuous latent, and a convolutional decoder that inverts back to audio (see the figure above):

• Semantic encoder (frozen). Reads an 80-bin mel spectrogram and captures content: the structure of what is being said or played, largely independent of fine acoustic variation.
• Acoustic encoder (trained). Reads a higher-resolution 160-bin mel spectrogram and captures fine spectral detail: the timbre and texture needed for a faithful waveform.
• Cross-attention fusion (trained). Merges the two streams into a single continuous latent.
• Decoder (trained). A ConvNeXt backbone with an inverse-STFT head reconstructs the waveform.

Keeping content and acoustic detail in separate streams lets the decoder preserve both, and makes the latent a semantic representation rather than just a compressed spectrum. Both mel spectrograms are computed inside the model, so it takes a raw waveform directly, with no external feature extraction.

Training runs in two stages. The semantic encoder is first pretrained on its own with masked spectrogram modeling, predicting masked time-frequency regions from the surrounding context to learn content from unlabeled audio. It is then frozen while the acoustic encoder, fusion, and decoder train jointly to reconstruct the waveform, so the acoustic stream learns only the detail the semantic stream leaves out.

How to get started

Install the dependencies:

pip install torch torchaudio torchcodec diffusers timm

Recent torchaudio releases decode audio through torchcodec, so torchaudio.load in the example below raises ImportError unless it is installed.

import torch
import torchaudio
from diffusers import AutoModel


def pad_for_model(waveform, hop_length=160, patch_multiple=16):
    """Pad to the next hop-aligned length whose frame count divides the patch size."""
    orig_len = waveform.shape[-1]
    frames = orig_len // hop_length + 1
    target_frames = -(-frames // patch_multiple) * patch_multiple
    target_len = hop_length * (target_frames - 1)
    if target_len < orig_len:
        target_len = hop_length * (target_frames + patch_multiple - 1)
    return torch.nn.functional.pad(waveform, (0, target_len - orig_len))


# The modeling code ships with the weights, so trust_remote_code=True is required.
model = AutoModel.from_pretrained("Motif-Technologies/Motif-Audio", trust_remote_code=True)
model = model.to("cuda", torch.bfloat16).eval()

waveform, sr = torchaudio.load("input.wav")
waveform = torchaudio.functional.resample(waveform, sr, 16000)  # model expects 16 kHz
waveform = waveform.mean(0, keepdim=True)                       # mono, (1, num_samples)

orig_len = waveform.shape[-1]
waveform = pad_for_model(waveform)               # frame count must divide the patch size
out = model(waveform.unsqueeze(0).to("cuda"))

reconstructed = out["waveform"][..., :orig_len]  # drop padding -> (1, 1, orig_len)

torchaudio.save("output.wav", reconstructed.squeeze(0).float().cpu(), 16000)

Cast to bfloat16 with .to(...) as shown. Passing torch_dtype=torch.bfloat16 to from_pretrained is not equivalent: it casts parameters but leaves the mel filterbank buffers in float32, which does not reproduce the reported scores.

forward returns a dict with four keys: waveform, the reconstructed 16 kHz audio, and the latents z_fused, z_sem, and z_acou (the fused and per-stream continuous representations, usable as downstream features).

Input requirements

• Mono, 16 kHz, shape (batch, 1, num_samples). Both mel spectrograms are computed inside the model, so no external feature extraction is needed.
• Run in bfloat16. The weights were trained under bfloat16 mixed precision; running the model in float32 measurably lowers reconstruction quality.
• Pad the input so the mel frame count is divisible by the encoder patch sizes (16 for the released model, the LCM of the semantic and acoustic temporal patch sizes), then trim the output back to the original length. The pad_for_model helper in the example above does exactly this.

Evaluation

Semantic understanding

Semantic scores come from frozen features with an MLP probe. Columns are accuracy unless the header names another metric, and higher is better throughout. In each column bold marks the best score and <ins>underline</ins> the second best.

Speech

Environment

Music

Reconstruction

Reconstruction quality across datasets. For each metric, the arrow marks the better direction.

Metrics:

• PESQ (Perceptual Evaluation of Speech Quality, ↑): perceptual quality of the reconstructed speech.
• STOI (Short-Time Objective Intelligibility, ↑): speech intelligibility.
• WER / CER (Word / Character Error Rate, ↓): ASR transcription error on the reconstructed audio; lower means content is better preserved. English WER is scored with Wav2Vec2 (facebook/wav2vec2-large-960h-lv60-self), Korean CER with Whisper-large-v3 (openai/whisper-large-v3).
• FAD (Fréchet Audio Distance, ↓): distance between reconstructed and reference audio, computed with the frechet-audio-distance library using VGGish embeddings.

Qualitative comparison

<p align="center"> <img src="assets/reconstruction_mel_comparison.png" alt="Log-mel spectrograms of an utterance reconstructed by X-Codec2, EnCodec, Mimi, DAC and Motif-Audio alongside the ground truth. A voiced region with strong harmonics is boxed and enlarged, and a third row shows the absolute difference from the reference" width="900"> </p>

The top row gives the full log-mel spectrogram of an utterance reconstructed by each system. The dashed box marks a voiced segment in which the harmonics are strong and well separated, and that region is enlarged in the middle row. The bottom row shows the absolute difference from the reference over the same region, so a darker panel indicates a closer reconstruction.

In the enlarged region, EnCodec flattens the harmonic peaks, reducing their depth relative to the surrounding valleys to roughly 60% of the reference, while the other systems preserve it. Motif-Audio's error panel is the darkest, showing the smallest deviation from the reference over that region.

Uses

Direct use

• Audio reconstruction and neural vocoding for general audio and speech at 16 kHz.
• Using the latent as an input representation for a downstream audio model.

Out-of-scope use

• Text-to-speech or other text-conditioned synthesis. The model reconstructs existing audio and accepts no text conditioning.
• Use as a discrete-token codec, or as a source of audio tokens for token-based language models. Its latent is continuous, not a sequence of quantized codebook indices, so it does not provide the discrete tokens those systems expect.

Bias, Risks, and Limitations

• Bandwidth ceiling. The model runs at 16 kHz, so it cannot represent content above the 8 kHz Nyquist limit. Uses that need full-band fidelity are out of scope.
• Lossy reconstruction. The output is a reconstruction, not a bit-exact copy. Subtle perceptual artifacts can remain even on in-distribution audio.
• Out-of-distribution inputs. Heavily degraded, very noisy, or non-audio inputs can produce audible artifacts.
• Potential for misuse. Because it reconstructs audio faithfully, it can serve as a component in voice-cloning or spoofing pipelines. Use it only on audio you have the right to process, and follow applicable law.

License

Released under the MIT License. You can use, modify, and redistribute the model and its weights, including commercially, as long as you keep the copyright and license notice. The model is provided as is, with no warranty.

Citation

@misc{motif_audio,
  title  = {Motif-Audio: A General-Purpose Audio Foundation Model},
  author = {Motif Technologies},
  year   = {2026}
}

Contact

For questions, use the Motif Technologies organization page on Hugging Face.