Uni-MoE-2.0-Omni

<h1 align="center">Uni-MoE 2.0-Omni</h1>

Uni-MoE 2.0 is a fully open-source omnimodal model that substantially advances the capabilities of Lychee's Uni-MoE series in language-centric multimodal understanding, reasoning, and generating. It is powered by Omnimodality 3D RoPE and Dynamic-Capacity Mixture-of-Experts architecture.

Uni-MoE 2.0-Omni is the version of the Uni-MoE 2.0 series that integrates full-modality understanding, as well as audio and image generation capabilities

<div align="center" style="display: flex; justify-content: center; margin-top: 10px;"> <a href="https://idealistxy.github.io/Uni-MoE-v2.github.io/"><img src="https://img.shields.io/badge/📰 -Website-228B22" style="margin-right: 5px;"></a> <a href="https://arxiv.org/abs/2511.12609"><img src="https://img.shields.io/badge/📄-Paper-8A2BE2" style="margin-right: 5px;"></a> <a href="https://github.com/HITsz-TMG/Uni-MoE"><img src="https://img.shields.io/badge/👨‍💻-Codes-007ACC" style="margin-right: 5px;"></a> </div>

---

If you enjoy our work or want timely updates, please give us a like and follow us.

Open-source Plan

• Model Checkpoint
  • Uni-MoE 2.0-Omni
  • Uni-MoE 2.0-Base
  • Uni-MoE 2.0-Thinking
  • Uni-MoE 2.0-Image
  • Uni-MoE 2.0-MoE-TTS
• Inference Code: HITsz-TMG/Uni-MoE-2.0
• Training Code: HITsz-TMG/Uni-MoE-2.0
• Technical Report: arxiv

Main Results

Model Introduction

<video controls playsinline width="100%" src="https://huggingface.co/HIT-TMG/Uni-MoE-2.0-Omni/resolve/main/imgs/audio.mp4"> </video> <video controls playsinline width="100%" src="https://huggingface.co/HIT-TMG/Uni-MoE-2.0-Omni/resolve/main/imgs/omni.mp4"> </video>

Getting Started

1. Clone this repository and navigate to the Uni-MoE 2.0 folder

git clone https://github.com/HITsz-TMG/Uni-MoE.git
cd Uni-MoE-2

2. Set up environment

Install the evaluation environment according to the requirements.

conda create -n uni_moe_2 python=3.11
conda activate uni_moe_2
pip install torch==2.5.1 torchaudio==2.5.1 torchvision==0.20.1
pip install -r requirements.txt
pip install flash-attn==2.6.0.post1 --no-build-isolation
pip install clip==1.0@git+https://github.com/openai/CLIP.git@dcba3cb2e2827b402d2701e7e1c7d9fed8a20ef1

Example Usage

We provide a simple example on the usage of this repo. For detailed usage, please refer to cookbook

import torch
from uni_moe.model.processing_qwen2_vl import Qwen2VLProcessor
from uni_moe.model.modeling_out import GrinQwen2VLOutForConditionalGeneration
from uni_moe.qwen_vl_utils import process_mm_info
from uni_moe.model import deepspeed_moe_inference_utils

processor = Qwen2VLProcessor.from_pretrained("HIT-TMG/Uni-MoE-2.0-Omni")

model = GrinQwen2VLOutForConditionalGeneration.from_pretrained("HIT-TMG/Uni-MoE-2.0-Omni", torch_dtype=torch.bfloat16).cuda()

processor.data_args = model.config

messages = [{
    "role": "user", 
    "content": [
            {"type": "text", "text": "<audio>\n<image>\nAnswer the question in the audio."},
            {"type": "audio", "audio": "examples/assets/audio/quick_start.mp3"},
            {"type": "image", "image": "examples/assets/image/quick_start.jpg"}
        ]
}]

texts = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
texts = texts.replace("<image>","<|vision_start|><|image_pad|><|vision_end|>").replace("<audio>","<|audio_start|><|audio_pad|><|audio_end|>").replace("<video>","<|vision_start|><|video_pad|><|vision_end|>")
image_inputs, video_inputs, audio_inputs = process_mm_info(messages)

inputs = processor(
    text=texts,
    images=image_inputs,
    videos=video_inputs,
    audios=audio_inputs,
    padding=True,
    return_tensors="pt",
)
inputs["input_ids"] = inputs["input_ids"].unsqueeze(0)

inputs = inputs.to(device=model.device)

output_ids = model.generate(
    **inputs,
    use_cache=True,
    pad_token_id=processor.tokenizer.eos_token_id,
    max_new_tokens=4096,
    temperature=1.0,
    do_sample=True
)

text = processor.batch_decode(output_ids[:, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)[0]
print(text)

<!-- # Citation Please cite the repo if you use the model or code in this repo. ``` ``` -->