LongCat-Flash-Omni-FP8

LongCat-Flash-Omni

<div align="center"> <img src="https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Omni/main/figures/LongCat-Flash-Omni.svg" width="300" alt="LongCat Logo"/> </div> <div align="center" style="line-height: 1;"> <a href="https://longcat.ai/" target="_blank" style="margin: 2px;"> <img alt="Omni" src="https://img.shields.io/badge/🤖%20Omni-LongCat--Flash--Omni-ADFF2F?color=29E154&logoColor=white" fill-opacity="1" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://github.com/meituan-longcat/LongCat-Flash-Omni"> <img alt="github" src="https://img.shields.io/badge/🤖%20Github-LongCat--Flash--Omni-ff6b6b?color=1783ff&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> </div> <div align="center" style="line-height: 1;"> <a href="https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Omni/main/figures/wechat_official_accounts.jpg" target="_blank" style="margin: 2px;"> <img alt="Wechat" src="https://img.shields.io/badge/WeChat-LongCat-brightgreen?logo=wechat&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://x.com/Meituan_LongCat" target="_blank" style="margin: 2px;"> <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-LongCat-white?logo=x&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> </div> <div align="center" style="line-height: 1;"> <a href="https://huggingface.co/meituan-longcat/LongCat-Flash-Omni-FP8/blob/main/LICENSE" style="margin: 2px;"> <img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/> </a> </div> <p align="center"> <a href="https://github.com/meituan-longcat/LongCat-Flash-Omni/blob/main/tech_report.pdf"><b>Tech Report</b>&nbsp;📄</a> </p>

Model Introduction

We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters (with 27B activated), excelling at real-time audio-visual interaction, which is attained by leveraging LongCat-Flash's high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, augmented by efficient multimodal perception and speech reconstruction modules. Through an effective curriculum-inspired progressive training strategy, our model achieves comprehensive multimodal capabilities while maintaining strong unimodal capability. Now, we open-source the model to foster future research and development in the community.

Model Architecture

<div align="center"> <img src="https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Omni/main/figures/longcat_flash_omni_architecture.png" width="60%" alt="LongCat-Flash-Omni" /> </div>

Key Features

🌟 SOTA and Unified Omni-Modal Model

LongCat-Flash-Omni is an open-source omni-modal model that achieves state-of-the-art cross-modal comprehension performance. It seamlessly integrates powerful offline multi-modal understanding with real-time audio–visual interaction within a single all-in-one framework.

🌟 Large-Scale with Low-Latency Audio–Visual Interaction

By leveraging an efficient LLM backbone, carefully designed lightweight modality encoders and decoder, and a chunk-wise audio–visual feature interleaving mechanism, LongCat-Flash-Omni achieves low-latency, high-quality audio–visual processing and streaming speech generation. It supports a context window of up to 128K tokens, enabling advanced capabilities in long-term memory, multi-turn dialogue, and temporal reasoning across multiple modalities.

🌟 Effective Early-Fusion Training

The model adopts an innovative multi-stage pretraining pipeline that progressively incorporates text, audio, and visual modalities under a balanced data strategy and early-fusion training paradigm, ensuring strong omni-modal performance without degradation in any single modality.

🌟 Efficient Training Infrastructure

Inspired by the concept of modality decoupling, we propose a Modality-Decoupled Parallelism training scheme that significantly enhances the efficiency of large-scale and highly challenging multimodal training.

🌟 Open-Source Contribution

We provide a comprehensive overview of the training methodology and data strategies behind LongCat-Flash-Omni, and release the model to accelerate future research and innovation in omni-modal intelligence.

For more detail, please refer to the comprehensive LongCat-Flash-Omni Technical Report.

Evaluation Results

<details open> <summary>Omni-modality</summary> </details> <details> <summary>Vision</summary>

Image-to-Text

Note: Values marked with * are sourced from public reports. As GPT-4o does not support image grounding, we do not report its results on RefCOCO and ScreenSpot-v2

---

Video-to-Text

Note: Values marked with * are sourced from public reports.

</details> <details> <summary>Audio</summary>

Table 1: Automatic Speech Recognition (ASR) and Speech-to-Text Translation (S2TT)

Note: ASR results are in CER/WER (lower is better), S2TT results are in BLEU score.

---

Table 2: Audio Understanding

---

Table 3: Audio-to-Text Chat

</details> <details> <summary>Text</summary>

Note: Values marked with * are sourced from other public reports. Note that DeepSeek-V3.1, Qwen3-235B-A22B, Gemini2.5-Flash, and Claude4-Sonnet are evaluated under their non-thinking mode.

</details>

Quick Start

Model Download

LongCat-Flash-Omni is a MoE model, which means that the model weights are distributed across multiple devices. Therefore, during loading in Hugging Face Transformers or vLLM, model weights will be automatically downloaded based on the model name. However, if your runtime environment is not conducive to downloading weights during execution, you can refer to the following commands to manually download the model weights to a local directory:

# Download through Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download meituan-longcat/LongCat-Flash-Omni --local-dir ./LongCat-Flash-Omni

Usage

We have implemented basic adaptations in SGLang to support running the Longcat-Flash-Omni model. Currently, the official SGLang does not natively support Longcat-Flash-Omni, so you can temporarily use our development branch for local installation and testing.

Due to its size of 560 billion parameters (560B), LongCat-Flash-Omni requires at least one node (e.g., 8×H20-141G) to host the model weights in FP8 format, and at least two nodes (e.g., 16×H800-80G) for BF16 weights. Detailed launch configurations are provided below.

Installation

• python >= 3.10.0 (Recommend to use Anaconda)
• PyTorch >= 2.8
• CUDA >= 12.9

conda create -n longcat python=3.10
conda activate longcat

# install SGLang
git clone -b longcat_omni_v0.5.3.post3 https://github.com/XiaoBin1992/sglang.git
pushd sglang
pip install -e "python"
popd

# install longcat-flash-omni demo
git clone https://github.com/meituan-longcat/LongCat-Flash-Omni
pushd LongCat-Flash-Omni
git submodule update --init --recursive
pip install -r requirements.txt
popd

Demo

The model can be served on your cluster using a combination of Tensor Parallelism and Expert Parallelism. Once all dependencies are installed, you can launch the demo using the following command.

• single-node inference

python3 longcat_omni_demo.py \
  --tp-size 8 \
  --ep-size 8 \
  --model-path where_you_download_model_dir \
  --output-dir output

• multi-node inference

python3 longcat_omni_demo.py \
  --tp-size 16 \
  --ep-size 16 \
  --nodes 2 \
  --node-rank $NODE_RANK \
  --dist-init-addr $MASTER_IP:5000 \
  --model-path where_you_download_model_dir \
  --output-dir output

NOTE: Replace $NODE_RANK and $MASTER_IP with the corresponding values of your GPU machines.

All test cases are defined in examples_dict.py, and additional test cases may be added as needed. After model execution, the generated results are saved in the directory specified by the --output-dir parameter.

Interaction with LongCat-Flash-Omni

Real-time Chat Website

You can use LongCat-Flash-Omni (web version currently only supports audio interaction features) on https://longcat.ai. The full service will be provided in subsequent updates.

APP

We are excited to announce that the LongCat-Flash-Omni app is now available for both Android and iOS.

For Android, you can download it from the following QR code.

<img src=https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Omni/main/figures/android_app_qrcode.jpg width="200px">

For iOS, you can download it by searching "LongCat" at App Store or QR code. Currently, only the Chinese App Store is supported.

<img src=https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Omni/main/figures/ios_app_qrcode.jpg width="200px">

License Agreement

The model weights are released under the MIT License.

Any contributions to this repository are licensed under the MIT License, unless otherwise stated. This license does not grant any rights to use Meituan trademarks or patents.

See the LICENSE file for the full license text.

Usage Considerations

This model has not been specifically designed or comprehensively evaluated for every possible downstream application.

Developers should take into account the known limitations of large language models, including performance variations across different languages, and carefully assess accuracy, safety, and fairness before deploying the model in sensitive or high-risk scenarios. It is the responsibility of developers and downstream users to understand and comply with all applicable laws and regulations relevant to their use case, including but not limited to data protection, privacy, and content safety requirements.

Nothing in this Model Card should be interpreted as altering or restricting the terms of the MIT License under which the model is released.

Citation

We kindly encourage citation of our work if you find it useful.

@misc{
    title={LongCat-Flash-Omni Technical Report}, 
    author={Meituan LongCat Team}, 
    year={2025}, 
    url={https://github.com/meituan-longcat/LongCat-Flash-Omni}, 
}

Contact

Please contact us at <a href="mailto:[email protected]">[email protected]</a> or join our WeChat Group if you have any questions.

WeChat Group

<img src=https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Omni/main/figures/wechat_qrcode.jpeg width="200px">