SpatioLM-Understanding-SenseNovaSI

SpatioLM-Understanding-SenseNovaSI

This is the official SpatioLM Understanding checkpoint based on SenseNova-SI 1.1 / InternVL3 (8B). It is intended for spatial reasoning and scene understanding.

SpatioLM adds a plug-and-play spatio-vision module to a frozen vision-language model and learns physically coherent representations from pseudo depth and camera-ray supervision. No additional 3D input is required at inference time.

Resources

• GitHub: https://github.com/xiaomi-research/spatio-lm
• Paper: https://arxiv.org/abs/2608.01899

Installation

git clone https://github.com/xiaomi-research/spatio-lm.git
cd spatio-lm
pip install -e .

Image inference

import torch
from lmms_eval.models.simple.internvl2 import load_image
from PIL import Image
from transformers import AutoTokenizer

from spatiolm.models import InternVL3RChatModel

checkpoint = "xiaomi-research/SpatioLM-Understanding-SenseNovaSI"
image = Image.open("/path/to/image.jpg").convert("RGB")

model = InternVL3RChatModel.from_pretrained(
    checkpoint,
    dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    use_fast=False,
)
pixel_values = load_image(image, input_size=448).to(
    device="cuda",
    dtype=torch.bfloat16,
)
answer = model.chat(
    tokenizer,
    pixel_values,
    "Which object is closer to the camera?",
    {"max_new_tokens": 128, "do_sample": False},
)
print(answer)

For video inference, benchmark evaluation, training details, and the Action checkpoint interface, see the SpatioLM repository.

Intended use and limitations

• This checkpoint is intended for research on physical spatial intelligence.
• Outputs can be inaccurate and should not be used as the sole signal in safety-critical or high-impact decisions.
• Performance can vary with image quality, viewpoint, scene domain, prompting, and video sampling strategy.
• The custom SpatioLM model implementation is required; loading with only stock Transformers auto classes is not supported.

Citation

@inproceedings{wu2026spatiolm,
  title={SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models},
  author={Wu, Jing and Wu, Jianhua and Guan, Jiayi and Chen, Jiahong and Lu, Jinghui and Ye, Hangjun and Gao, Bingzhao and Chen, Long},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026},
  note={To appear},
  eprint={2608.01899},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2608.01899}
}