Ming-UniAudio-16B-A3B-Edit

Ming-UniAudio

<p align="center">📑 <a href="">Technical Report</a>｜📖<a href="https://xqacmer.github.io/Ming-Unitok-Audio.github.io/">Project Page</a> ｜🤗 <a href="https://huggingface.co/inclusionAI/Ming-UniAudio-16B-A3B-Edit">Hugging Face</a>｜ 🤖 <a href="https://modelscope.cn/models/inclusionAI/Ming-UniAudio-16B-A3B-Edit">ModelScope</a>

Introduction

Ming-UniAudio is a novel framework that unifies speech understanding, generation, and editing. Its core is a unified continuous speech tokenizer that effectively unifies semantic and acoustic features within an end-to-end model. We developed a speech language model that strikes a balance between generation and understanding capabilities based on the unified continuous audio tokenizer. Leveraging this foundational model, which exhibits robust performance in both domains, we further trained a dedicated speech editing model built upon Ming-Lite-Omni. Crucially, Ming-UniAudio is the first to enable universal, free-form speech editing guided solely by natural language instructions, handling complex semantic and acoustic modifications without manual region specification.

• 🔥 First unified continuous speech tokenizer for both understanding and generation tasks: MingTok-Audio
• 🔥 First Speech LLM with unifed continuous tokenizer for both understanding and generation: Ming-UniAudio
• 🔥 First universal free-form speech editing model for various semantic and acoustic editing task without any temporal regime: Ming-UniAudio-Edit
• 🔥 First benchmark for free-form speech editing: Ming-Freeform-Audio-Edit-Benchmark

📌 Updates

• [2025.09.30] 🔥 We release Ming-UniAudio with significant improvements across speech understanding, generation, and free-form editing tasks.

Key Features

Ming-UniAudio features key optimizations as follows, compared to other audio-assisted LLMs:

• Unified Continuous Speech Tokenizer: Ming-UniAudio proposes a unified continuous speech tokenizer MingTok-Audio based on a VAE framework with a causal Transformer architecture, the first continuous speech tokenizer to effectively integrate semantic and acoustic features, and enables a closed-loop system with LLMs through hierarchical feature representations, makes it suitable for both understanding and generation tasks
• Unified Speech Language Model for Generation and Understanding: We pretrain an end-to-end unified speech language model with a single LLM backbone for both understanding and generation tasks, enhanced with a Diffusion Head to ensure high-fidelity speech synthesis.
• Instruction-Guided Free-Form Speech Editing: We introduce the first instruction-guided, free-form speech editing framework that supports comprehensive semantic and acoustic edits without requiring explicit edit regions, along with Ming-Freeform-Audio-Edit, the first open-source evaluation set for such tasks.

<!-- <p align="center"> <img src="./figures/uniaudio-tokenizer.pdf" width="600"/> <p> -->

Evaluation

In various benchmark tests, Ming-UniAudio demonstrates highly competitive results compared to industry-leading models of similar scale.

Speech Understanding

<table> <caption>Context ASR performance comparison on various audio benchmark datasets.</caption> <thead> <tr> <th rowspan="2"><strong>Datasets</strong></th> <th rowspan="2"><strong>Model</strong></th> <th colspan="4"><strong>Performance</strong></th> </tr> <tr> <th> <strong>Speech-English</strong><br> <small><strong>WER | NE-WER | NE-FNR</strong></small> </th> <th> <strong>Dialogue-English</strong><br> <small><strong>WER | NE-WER | NE-FNR</strong></small> </th> <th> <strong>Speech-Mandarin</strong><br> <small><strong>WER | NE-WER | NE-FNR</strong></small> </th> <th> <strong>Dialogue-Mandarin</strong><br> <small><strong>WER | NE-WER | NE-FNR</strong></small> </th> </tr> </thead> <tbody> <tr> <td rowspan="7"> <strong>Understanding</strong> <br> <strong>Context ASR</strong><br> </td> <td>Qwen2-Audio</td> <td>11.49 | 27.27 | 35.08</td> <td>13.99 | 33.02 | 32.92</td> <td>9.92 | 24.10 | 30.02</td> <td>7.00 | 22.76 | 26.17</td> </tr> <tr> <td>Baichuan-Audio</td> <td>7.52 | 5.87 | 4.55</td> <td>5.66 | 10.01 | 3.64</td> <td>2.16 | 6.65 | <strong>2.35</strong></td> <td>2.96 | 11.48 | 3.94</td> </tr> <tr> <td>Kimi-Audio</td> <td><strong>2.90</strong> | 6.68 | 8.01</td> <td><strong>4.67</strong> | 13.50 | 11.31</td> <td>1.95 | 11.13 | 15.28</td> <td>2.90 | 15.91 | 16.68</td> </tr> <tr> <td>Baichuan-Omni-1.5</td> <td>8.16 | 7.69 | 6.53</td> <td>9.91 | 14.40 | 5.54</td> <td>2.98 | 8.39 | 4.71</td> <td>5.00 | 16.83 | 7.84</td> </tr> <tr> <td>Qwen2.5-Omni-3B</td> <td>3.99 | 7.80 | 9.69</td> <td>4.83 | 14.36 | 12.85</td> <td>2.13 | 10.55 | 14.11</td> <td>3.12 | 15.07 | 15.17</td> </tr> <tr> <td>Qwen2.5-Omni-7B</td> <td>3.96 | 7.38 | 8.72</td> <td>5.32 | 11.83 | 9.24</td> <td>1.84 | 9.80 | 12.19</td> <td><strong>2.40</strong> | 14.06 | 13.17</td> </tr> <tr> <td><strong>Ming-UniAudio-16B-A3B-Edit(ours)</strong></td> <td>4.00 | <strong>3.56</strong> | <strong>3.69</strong></td> <td>5.34 | <strong>8.73</strong> | <strong>2.53</strong></td> <td><strong>1.58</strong> | <strong>5.98</strong> | 2.40</td> <td>3.04 | <strong>9.50</strong> | <strong>1.48</strong></td> </tr> </tbody> </table>

Speech Editing

<table> <caption>Performance on various audio benchmark datasets.</caption> <thead> <tr> <th rowspan="2"><strong>Datasets</strong></th> <th rowspan="2"><strong>Model</strong></th> <th colspan="4"><strong>Performance</strong></th> </tr> </thead> <tbody> <tr> <th> <strong>Deletion-basic<br> <strong>Deletion</strong> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 11.89 | 14.85<br> 22.92 | 27.60 </td> <td align="center"> <strong>ACC zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 100 | 82.22<br> 82.92 | 85 </td> <td align="center"> <strong>SIM zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.78 | 0.76<br> 0.81 | 0.74 </td> <td align="center"> <strong>no-edit WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 11.49 | 24.26<br> 17.50 | 35.21 </td> </tr> <tr> <th> <strong>Insertion-basic<br> <strong>Insertion</strong> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 3.42 | 6.63<br> 3.89 | 7.592 </td> <td align="center"> <strong>ACC zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 80 | 71.43<br> 79.31 | 62.31 </td> <td align="center"> <strong>SIM zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.83 | 0.79<br> 0.83 | 0.79 </td> <td align="center"> <strong>no-edit WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 3.52 | 17.70<br> 4.10 | 18.84 </td> </tr> <tr> <th> <strong>Substitution-basic<br> <strong>Substitution</strong> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 4.52 | 8.99<br> 4.56 | 7.64 </td> <td align="center"> <strong>ACC zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 78.62 | 59.78<br> 76.62 | 65.62 </td> <td align="center"> <strong>SIM zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.82 | 0.78<br> 0.83 | 0.77 </td> <td align="center"> <strong>no-edit WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 4.63 | 19.28<br> 4.75 | 18.39 </td> </tr> <tr> <th> <strong>Dialect Conversion<br> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%)</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 8.93 </td> <td align="center"> <strong>ACC</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.50 </td> <td align="center"> <strong>SIM</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.66 </td> <td align="center"> - </td> </tr> <tr> <th> <strong>Speed changing<br> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 5.88 | 17.53 </td> <td align="center"> <strong>SIM zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.66 | 0.57 </td> <td align="center"> <strong>RDE(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 6.36 | 5.92 </td> <td align="center"> - </td> </tr> <tr> <th> <strong>Pitch changing<br> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 7.45 | 13.37 </td> <td align="center"> <strong>SIM zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.36 | 0.24 </td> <td align="center"> - </td> <td align="center"> - </td> </tr> <tr> <th> <strong>Volume changing<br> </th> <td>Ming-UniAudio-16B-A3B-Edit</td> <td align="center"> <strong>WER(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 1.71 | 1.35 </td> <td align="center"> <strong>SIM zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 0.86 | 0.80 </td> <td align="center"> <strong>RAE(%) zh | en</strong><br> <hr style="height: 1px; background-color: black; border: none; margin: 2px 0;"> 14.9 | 11.7 </td> <td align="center"> - </td> </tr> </tbody> </table> <!-- Denoise Group --> <h4>Denoise</h4> <table> <caption>Performance comparison on various audio benchmark datasets. The best results are in <strong>bold</strong>.</caption> <thead> <tr> <th><strong>Datasets</strong></th> <th><strong>Model</strong></th> <th><strong>Model Type</strong></th> <th><strong>DNSMOS OVRL</strong></th> <th><strong>DNSMOS SIG</strong></th> <th><strong>DNSMOS BAK</strong></th> </tr> </thead> <tbody> <tr> <td rowspan="8"><strong>Denoise</strong></td> <td>FullSubNet</td> <td rowspan="6">specialized</td> <td>2.93</td> <td>3.05</td> <td>3.51</td> </tr> <tr> <td>Inter-Subnet</td> <td>2.98</td> <td>3.17</td> <td>3.15</td> </tr> <tr> <td>CDiffuSE</td> <td>2.84</td> <td>3.37</td> <td>3.52</td> </tr> <tr> <td>SGMSE</td> <td>3.11</td> <td>3.47</td> <td>3.41</td> </tr> <tr> <td>StoRM</td> <td>3.15</td> <td>3.54</td> <td>3.69</td> </tr> <tr> <td>GenSE</td> <td><strong>3.43</strong></td> <td><strong>3.65</strong></td> <td><strong>4.18</strong></td> </tr> <tr> <td>MiMo-Audio</td> <td rowspan="2">general</td> <td>3.30</td> <td>3.56</td> <td>4.10</td> </tr> <tr> <td><strong>Ming-UniAudio-16B-A3B-Edit(ours)</strong></td> <td>3.26</td> <td>3.59</td> <td>3.97</td> </tr> </tbody> </table>

Model & Benchmark Downloads

You can download our latest model and Benchmark from both Huggingface and ModelScope.

<div align="center"> </div> If you're in mainland China, we strongly recommend you to download our model from 🤖 <a href="https://modelscope.cn/models/inclusionAI/Ming-UniAudio-16B-A3B">ModelScope</a>.

pip install modelscope
modelscope download --model inclusionAI/Ming-UniAudio-16B-A3B-Edit --local_dir inclusionAI/Ming-UniAudio-16B-A3B-Edit  --revision master

Note: This download process will take several minutes to several hours, depending on your network conditions.

Use Cases

Additional demonstration cases are available on our project page.

Environment Preparation

Installation with pip

pip install -r requirements.txt

Installation with docker

You can also initialize the environment by building the docker image. First clone this repository:

git clone --depth 1 https://github.com/inclusionAI/Ming-UniAudio
cd Ming-UniAudio

Then build the docker image with the provided Dockerfile in docker/docker-py310-cu121. This step might take a while:

docker build -t ming:py310-cu121 docker/docker-py310-cu121

At last, start the container with the current repo directory mounted:

docker run -it --gpus all -v "$(pwd)":/workspace/Ming-UniAudio ming:py310-cu121 ming:py310-cu121 /bin/bash

You can run the model with python interface. You may download the huggingface model in the repo directory first (.../Ming-UniAudio/) or mount the downloaded model path when starting the container.

Example Usage

We provide a step-by-step running example:

Step 1 - Download the source code

git clone	https://github.com/inclusionAI/Ming-UniAudio
cd Ming-UniAudio

Step 2 - Download the Ming-UniAudio model weights and create a soft link to the source code directory

Download our model following Model & Benchmark Downloads

mkdir inclusionAI 
ln -s /path/to/inclusionAI/Ming-UniAudio-16B-A3B-Edit inclusionAI/Ming-UniAudio-16B-A3B-Edit

Step 3 - Enter the code directory, you can refer to the following codes to run the Ming-UniAudio model.

jupyter notebook cookbooks/demo.ipynb

We also provide a simple example on the usage of this repo. For detailed usage, please refer to demobook.ipynb.

import warnings
import torch
from transformers import AutoProcessor

from modeling_bailingmm import BailingMMNativeForConditionalGeneration

import random
import numpy as np
from loguru import logger

def seed_everything(seed=1895):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

seed_everything()
warnings.filterwarnings("ignore")

class MingAudio:
    def __init__(self, model_path, device="cuda:0"):
        self.device = device
        self.model = BailingMMNativeForConditionalGeneration.from_pretrained(
            model_path,
            torch_dtype=torch.bfloat16,
            low_cpu_mem_usage=True,
        ).eval().to(torch.bfloat16).to(self.device)
        self.processor = AutoProcessor.from_pretrained(".", trust_remote_code=True)
        self.tokenizer = self.processor.tokenizer
        self.sample_rate = self.processor.audio_processor.sample_rate
        self.patch_size = self.processor.audio_processor.patch_size
        
    def speech_understanding(self, messages):
        text = self.processor.apply_chat_template(messages, add_generation_prompt=True)
        image_inputs, video_inputs, audio_inputs = self.processor.process_vision_info(messages)

        inputs = self.processor(
            text=[text],
            images=image_inputs,
            videos=video_inputs,
            audios=audio_inputs,
            return_tensors="pt",
        ).to(self.device)

        for k in inputs.keys():
            if k == "pixel_values" or k == "pixel_values_videos" or k == "audio_feats":
                inputs[k] = inputs[k].to(dtype=torch.bfloat16)
        logger.info(f"input: {self.tokenizer.decode(inputs['input_ids'].cpu().numpy().tolist()[0])}")

        generated_ids = self.model.generate(
            **inputs,
            max_new_tokens=512,
            eos_token_id=self.processor.gen_terminator,
        )
        generated_ids_trimmed = [
            out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
        ]
        output_text = self.processor.batch_decode(
            generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
        )[0]

        return output_text

    def speech_generation(
        self, 
        text,
        prompt_wav_path,
        prompt_text,
        lang='zh',
        output_wav_path='out.wav'
    ):
        waveform = self.model.generate_tts(
            text=text,
            prompt_wav_path=prompt_wav_path,
            prompt_text=prompt_text,
            patch_size=self.patch_size,
            tokenizer=self.tokenizer,
            lang=lang,
            output_wav_path=output_wav_path,
            sample_rate=self.sample_rate,
            device=self.device
        )
        
        return waveform

    def speech_edit(
        self, 
        messages,
        output_wav_path='out.wav'
    ):
        text = self.processor.apply_chat_template(messages, add_generation_prompt=True)
        image_inputs, video_inputs, audio_inputs = self.processor.process_vision_info(messages)

        inputs = self.processor(
            text=[text],
            images=image_inputs,
            videos=video_inputs,
            audios=audio_inputs,
            return_tensors="pt",
        ).to(self.device)

        ans = torch.tensor([self.tokenizer.encode('<answer>')]).to(inputs['input_ids'].device)
        inputs['input_ids'] = torch.cat([inputs['input_ids'], ans], dim=1)
        attention_mask = inputs['attention_mask']
        inputs['attention_mask'] = torch.cat((attention_mask, attention_mask[:, :1]), dim=-1)
        for k in inputs.keys():
            if k == "pixel_values" or k == "pixel_values_videos" or k == "audio_feats":
                inputs[k] = inputs[k].to(dtype=torch.bfloat16)
        logger.info(f"input: {self.tokenizer.decode(inputs['input_ids'].cpu().numpy().tolist()[0])}")

        edited_speech, edited_text = self.model.generate_edit(
            **inputs,
            tokenizer=self.tokenizer,
            output_wav_path=output_wav_path
        )
        return edited_speech, edited_text

if __name__ == "__main__":
    model = MingAudio("inclusionAI/Ming-UniAudio-16B-A3B-Edit")

    # Edit
    messages = [
        {
            "role": "HUMAN",
            "content": [
                {"type": "audio", "audio": "data/wavs/00004768-00000024.wav", "target_sample_rate": 16000},
                {
                    "type": "text",
                    "text": "<prompt>Please recognize the language of this speech and transcribe it. And insert '实现' before the character or word at index 3.\n</prompt>",
                },
            ],
        },
    ]
    
    response = model.speech_edit(messages=messages)
    logger.info(f"Generated Response: {response}")

Note: We test the examples on hardware of NVIDIA H800-80GB/H20-96G with CUDA 12.4.

Citation

If you find our work helpful, feel free to give us a cite.