Ming-UniAudio-16B-A3B

Ming-UniAudio

<p align="center">📑 <a href="https://mdn.alipayobjects.com/cto_asrtts/uri/file/as/TR-Ming-UniAudio.pdf">Technical Report</a>｜📖<a href="https://xqacmer.github.io/Ming-Unitok-Audio.github.io/">Project Page</a> ｜🤗 <a href="https://huggingface.co/inclusionAI/Ming-UniAudio-16B-A3B">Hugging Face</a>｜ 🤖 <a href="https://modelscope.cn/models/inclusionAI/Ming-UniAudio-16B-A3B">ModelScope</a>

Introduction

Ming-UniAudio is a novel framework that unifies speech understanding, generation, and editing. Its core is a unified continuous speech tokenizer that effectively unifies semantic and acoustic features within an end-to-end model. We developed a speech language model that strikes a balance between generation and understanding capabilities based on the unified continuous audio tokenizer. Leveraging this foundational model, which exhibits robust performance in both domains, we further trained a dedicated speech editing model built upon Ming-Lite-Omni. Crucially, Ming-UniAudio is the first to enable universal, free-form speech editing guided solely by natural language instructions, handling complex semantic and acoustic modifications without manual region specification.

• 🔥 First unified continuous speech tokenizer for both understanding and generation tasks: MingTok-Audio
• 🔥 First Speech LLM with unifed continuous tokenizer for both understanding and generation: Ming-UniAudio
• 🔥 First universal free-form speech editing model for various semantic and acoustic editing task without timestamp condition: Ming-UniAudio-Edit
• 🔥 First benchmark for free-form speech editing: Ming-Freeform-Audio-Edit-Benchmark

<p align="center"> <img src="./figures/uniaudio.png" width="600"/> <p>

📌 Updates

• [2025.09.30] 🔥 We release Ming-UniAudio with significant improvements across speech understanding, generation, and free-form editing tasks.

Key Features

Ming-UniAudio features key optimizations as follows, compared to other audio-assisted LLMs:

• Unified Continuous Speech Tokenizer: Ming-UniAudio proposes a unified continuous speech tokenizer MingTok-Audio based on a VAE framework with a causal Transformer architecture, the first continuous speech tokenizer to effectively integrate semantic and acoustic features, and enables a closed-loop system with LLMs through hierarchical feature representations, makes it suitable for both understanding and generation tasks

• Unified Speech Language Model for Generation and Understanding: We pretrain an end-to-end unified speech language model with a single LLM backbone for both understanding and generation tasks, enhanced with a Diffusion Head to ensure high-fidelity speech synthesis.

• Instruction-Guided Free-Form Speech Editing: We introduce the first instruction-guided, free-form speech editing framework that supports comprehensive semantic and acoustic edits without requiring explicit edit regions, along with Ming-Freeform-Audio-Edit, the first open-source evaluation set for such tasks.

<!-- <p align="center"> <img src="./figures/uniaudio-tokenizer.pdf" width="600"/> <p> -->

Evaluation

In various benchmark tests, Ming-UniAudio demonstrates highly competitive results compared to industry-leading models of similar scale.

Speech Understanding

<table> <caption>ASR performance comparison on various audio benchmark datasets. The best results are in <strong>bold</strong>.</caption> <thead> <tr> <th rowspan="2"><strong>Datasets</strong></th> <th rowspan="2"><strong>Model</strong></th> <th colspan="7"><strong>Performance</strong></th> </tr> <tr> <th><strong>aishell2-ios</strong></th> <th><strong>LS-clean</strong></th> <th><strong>Hunan</strong></th> <th><strong>Minnan</strong></th> <th><strong>Guangyue</strong></th> <th><strong>Chuanyu</strong></th> <th><strong>Shanghai</strong></th> </tr> </thead> <tbody> <tr> <td rowspan="4"><strong>Understanding ASR</strong></td> <td>Kimi-Audio</td> <td><strong>2.56</strong></td> <td><strong>1.28</strong></td> <td>31.93</td> <td>80.28</td> <td>41.49</td> <td>6.69</td> <td>60.64</td> </tr> <tr> <td>Qwen2.5 Omni</td> <td>2.75</td> <td>1.80</td> <td>29.31</td> <td>53.43</td> <td>10.39</td> <td>7.61</td> <td>32.05</td> </tr> <tr> <td>Qwen2 Audio</td> <td>2.92</td> <td>1.60</td> <td>25.88</td> <td>123.78</td> <td>7.59</td> <td>7.77</td> <td>31.73</td> </tr> <tr> <td><strong>Ming-UniAudio-16B-A3B(ours)</strong></td> <td>2.84</td> <td>1.62</td> <td><strong>9.80</strong></td> <td><strong>16.50</strong></td> <td><strong>5.51</strong></td> <td><strong>5.46</strong></td> <td><strong>14.65</strong></td> </tr> </tbody> </table>

Speech Generation

<table align="center"> <caption>Performance comparison on various audio benchmark datasets. The best results are in <strong>bold</strong>.</caption> <thead> <tr> <th align="left"><b>Datasets</b></th> <th align="left"><b>Model</b></th> <th colspan="4" align="center"><b>Performance</b></th> </tr> <tr> <th></th> <th></th> <th align="center"><b>Seed-zh WER(%)</b></th> <th align="center"><b>Seed-zh SIM</b></th> <th align="center"><b>Seed-en WER(%)</b></th> <th align="center"><b>Seed-en SIM</b></th> </tr> </thead> <tbody> <tr> <td rowspan="5" align="left" style="vertical-align: middle;"><b>Generation</b></td> <td align="left">Seed-TTS</td> <td align="center">1.12</td> <td align="center"><b>0.80</b></td> <td align="center">2.25</td> <td align="center"><b>0.76</b></td> </tr> <tr> <td align="left">MiMo-Audio</td> <td align="center">1.96</td> <td align="center">-</td> <td align="center">5.37</td> <td align="center">-</td> </tr> <tr> <td align="left">Qwen3-Omni-30B-A3B-Instruct</td> <td align="center">1.07</td> <td align="center">-</td> <td align="center"><b>1.39</b></td> <td align="center">-</td> </tr> <tr> <td align="left">Ming-Omni-Lite</td> <td align="center">1.69</td> <td align="center">0.68</td> <td align="center">4.31</td> <td align="center">0.51</td> </tr> <tr> <td align="left"><strong>Ming-UniAudio-16B-A3B(ours)</strong></td> <td align="center"><b>0.95</b></td> <td align="center">0.70</td> <td align="center">1.85</td> <td align="center">0.58</td> </tr> </tbody> </table>

Model & Benchmark Downloads

You can download our latest model and Benchmark from both Huggingface and ModelScope.

<div align="center"> </div> If you're in mainland China, we strongly recommend you to download our model from 🤖 <a href="https://modelscope.cn/models/inclusionAI/Ming-UniAudio-16B-A3B">ModelScope</a>.

pip install modelscope
modelscope download --model inclusionAI/Ming-UniAudio-16B-A3B --local_dir inclusionAI/Ming-UniAudio-16B-A3B  --revision master

Note: This download process will take several minutes to several hours, depending on your network conditions.

Use Cases

Additional demonstration cases are available on our project page.

Environment Preparation

Installation with pip

pip install -r requirements.txt

Installation with docker

You can also initialize the environment by building the docker image. First clone this repository:

git clone --depth 1 https://github.com/inclusionAI/Ming-UniAudio
cd Ming-UniAudio

Then build the docker image with the provided Dockerfile in docker/docker-py310-cu121. This step might take a while:

docker build -t ming:py310-cu121 docker/docker-py310-cu121

At last, start the container with the current repo directory mounted:

docker run -it --gpus all -v "$(pwd)":/workspace/Ming-UniAudio ming:py310-cu121 ming:py310-cu121 /bin/bash

You can run the model with python interface. You may download the huggingface model in the repo directory first (.../Ming-UniAudio/) or mount the downloaded model path when starting the container.

Example Usage

We provide a step-by-step running example:

Step 1 - Download the source code

git clone	https://github.com/inclusionAI/Ming-UniAudio
cd Ming-UniAudio

Step 2 - Download the Ming-UniAudio model weights and create a soft link to the source code directory

Download our model following Model & Benchmark Downloads

mkdir inclusionAI 
ln -s /path/to/inclusionAI/Ming-UniAudio-16B-A3B inclusionAI/Ming-UniAudio-16B-A3B

Step 3 - Enter the code directory, you can refer to the following codes to run the Ming-UniAudio model.

jupyter notebook cookbooks/demo.ipynb

We also provide a simple example on the usage of this repo. For detailed usage, please refer to demobook.ipynb.

import warnings
import torch
from transformers import AutoProcessor

from modeling_bailingmm import BailingMMNativeForConditionalGeneration

import random
import numpy as np
from loguru import logger

def seed_everything(seed=1895):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

seed_everything()
warnings.filterwarnings("ignore")

class MingAudio:
    def __init__(self, model_path, device="cuda:0"):
        self.device = device
        self.model = BailingMMNativeForConditionalGeneration.from_pretrained(
            model_path,
            torch_dtype=torch.bfloat16,
            low_cpu_mem_usage=True,
        ).eval().to(torch.bfloat16).to(self.device)
        self.processor = AutoProcessor.from_pretrained(".", trust_remote_code=True)
        self.tokenizer = self.processor.tokenizer
        self.sample_rate = self.processor.audio_processor.sample_rate
        self.patch_size = self.processor.audio_processor.patch_size
        
    def speech_understanding(self, messages):
        text = self.processor.apply_chat_template(messages, add_generation_prompt=True)
        image_inputs, video_inputs, audio_inputs = self.processor.process_vision_info(messages)

        inputs = self.processor(
            text=[text],
            images=image_inputs,
            videos=video_inputs,
            audios=audio_inputs,
            return_tensors="pt",
        ).to(self.device)

        for k in inputs.keys():
            if k == "pixel_values" or k == "pixel_values_videos" or k == "audio_feats":
                inputs[k] = inputs[k].to(dtype=torch.bfloat16)
        logger.info(f"input: {self.tokenizer.decode(inputs['input_ids'].cpu().numpy().tolist()[0])}")

        generated_ids = self.model.generate(
            **inputs,
            max_new_tokens=512,
            eos_token_id=self.processor.gen_terminator,
        )
        generated_ids_trimmed = [
            out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
        ]
        output_text = self.processor.batch_decode(
            generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
        )[0]

        return output_text

    def speech_generation(
        self, 
        text,
        prompt_wav_path,
        prompt_text,
        lang='zh',
        output_wav_path='out.wav'
    ):
        waveform = self.model.generate_tts(
            text=text,
            prompt_wav_path=prompt_wav_path,
            prompt_text=prompt_text,
            patch_size=self.patch_size,
            tokenizer=self.tokenizer,
            lang=lang,
            output_wav_path=output_wav_path,
            sample_rate=self.sample_rate,
            device=self.device
        )
        
        return waveform

    def speech_edit(
        self, 
        messages,
        output_wav_path='out.wav'
    ):
        text = self.processor.apply_chat_template(messages, add_generation_prompt=True)
        image_inputs, video_inputs, audio_inputs = self.processor.process_vision_info(messages)

        inputs = self.processor(
            text=[text],
            images=image_inputs,
            videos=video_inputs,
            audios=audio_inputs,
            return_tensors="pt",
        ).to(self.device)

        ans = torch.tensor([self.tokenizer.encode('<answer>')]).to(inputs['input_ids'].device)
        inputs['input_ids'] = torch.cat([inputs['input_ids'], ans], dim=1)
        attention_mask = inputs['attention_mask']
        inputs['attention_mask'] = torch.cat((attention_mask, attention_mask[:, :1]), dim=-1)
        for k in inputs.keys():
            if k == "pixel_values" or k == "pixel_values_videos" or k == "audio_feats":
                inputs[k] = inputs[k].to(dtype=torch.bfloat16)
        logger.info(f"input: {self.tokenizer.decode(inputs['input_ids'].cpu().numpy().tolist()[0])}")

        edited_speech, edited_text = self.model.generate_edit(
            **inputs,
            tokenizer=self.tokenizer,
            output_wav_path=output_wav_path
        )
        return edited_speech, edited_text

if __name__ == "__main__":
    model = MingAudio("inclusionAI/Ming-UniAudio-16B-A3B")
    
    # ASR
    messages = [
        {
            "role": "HUMAN",
            "content": [
                {
                    "type": "text",
                    "text": "Please recognize the language of this speech and transcribe it. Format: oral.",
                },
                
                {"type": "audio", "audio": "data/wavs/BAC009S0915W0292.wav"},
            ],
        },
    ]
    
    response = model.speech_understanding(messages=messages)
    logger.info(f"Generated Response: {response}")

    # TTS
    model.speech_generation(
        text='我们的愿景是构建未来服务业的数字化基础设施，为世界带来更多微小而美好的改变。',
        prompt_wav_path='data/wavs/10002287-00000094.wav',
        prompt_text='在此奉劝大家别乱打美白针。',
    )

Note: We test the examples on hardware of NVIDIA H800-80GB/H20-96G with CUDA 12.4.

Citation

If you find our work helpful, feel free to give us a cite.