Spatial-Omni

Spatial-Omni

Checkpoints for Spatial-Omni, a spatial-audio extension using a dedicated FOA encoder.

Paper · Code

Models

The 7B checkpoints contain the trained spatial encoder, projector and LLM LoRA parameters. MIX additionally includes learned null tokens. Use them with the original base model and Spatial-Omni code; they are not standalone Transformers model repositories.

Usage

Each 7B variant has its own train_args.json; keep it beside that variant's checkpoint.

Base model: Qwen/Qwen2.5-Omni-7B. Before running the code, set model_id to your local base-model path and beats_checkpoint to the absolute path of the downloaded SO-Encoder.pt. The encoder path in the supplied JSON is relative to that JSON's directory; resolve it when preparing local settings. Follow the inference instructions in the code repository.

Input follows the released dataset's four-channel FOA convention at 16 kHz. Preserve channel order, with W in channel 0. These checkpoints use a 20-second spatial window and 2.5-Hz projected spatial tokens.

Recorded results

SO-Bench uses 7,877 examples, MMAU-mini 1,000, and MMAU-Pro 4,163 unique examples.

License and citation

Checkpoints are licensed under CC BY-NC-SA 4.0. Upstream base models and third-party code retain their respective licenses.

Please cite Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding, arXiv:2606.10738.

@article{zhu2026spatial,
  title={Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding},
  author={Zhu, Zhiyuan and Chen, Yixuan and Shao, Yiwen and Guo, Wenxiang and Pan, Changhao and Zhang, Yu and Wang, Yuxiang and Liu, Wei and Zhang, Houhua and Zeng, Chengkuan and others},
  journal={arXiv preprint arXiv:2606.10738},
  year={2026}
}