SIS-Motion

choucsan/SIS-Motion

SIS-Motion is a video-text-to-text model from choucsan. Use it for the video-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.

<p align="center" <img src="images/poster.jpeg" alt="SIS-Motion" width="900"/ </p <p align="center" <a href="https://arxiv.org/abs/2607.12477"<img src="https://img.shields.io/badge/arXiv-2607.12477-b31b1b?style=for-th…

Downloads · 30 days

21% of all-time downloads

All-time downloads

171

Public

Repo size

126 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.pth54.1 MB · 43%

At a glance

Task: Video-Text-to-Text
Library: peft
License: apache-2.0
Access: Public
Created: Jul 12, 2026
Updated: Jul 18, 2026
SHA: 0f11f457

Base models

Qwen/Qwen2.5-VL-7B-Instruct

Model card

From the Hugging Face model README

SIS-Motion

SIS-Motion is a motion-aware extension of a standard video MLLM, designed to investigate whether explicitly modeling self-related dynamics can improve the joint understanding of space and self in embodied UAV scenarios.

Important: SIS-Motion is NOT a state-of-the-art model. It is a controlled experiment where the only difference from the visual-only backbone is the addition of a frozen optical-flow estimator, a motion-feature branch, and a lightweight fusion connector. All other settings — base model, training data, LoRA rank, learning rate, and optimization — are kept identical. This ensures that any observed gain can be cleanly attributed to motion-aware integration rather than differences in scale or recipe.

Architecture

SIS-Motion extends Qwen2.5-VL with a motion-aware branch. Video frames are processed in parallel through a shared vision encoder — once for appearance and once for optical-flow-derived motion features — then fused via a lightweight additive connector before the language model.

The forward path is:

video frames
├── Qwen2.5-VL vision encoder (frozen) ───────────────> visual tokens
└── VideoFlow MOFNet (frozen, FP32)
    └── forward/backward flow [T, 4, H/8, W/8]
        └── forward flow (dx, dy)
            └── pseudo-images [magnitude, dx, dy]
                └── Qwen2.5-VL vision encoder (shared, frozen) -> motion tokens

motion tokens -> LayerNorm -> Linear(3584, 2048) -> GELU
              -> Linear(2048, 3584)
visual tokens + projected motion tokens -> Qwen2.5-VL language model with LoRA

1. Appearance branch

The standard Qwen2.5-VL video processor and frozen vision encoder produce the appearance tokens used by the base model.

2. Motion branch

VideoFlow MOFNet receives at least three RGB frames, resizes them to 320x480, and predicts forward and backward optical flow at 1/8 resolution. SIS-Motion uses the forward (dx, dy) channels, clamps invalid/extreme values, and builds a three-channel pseudo-image [magnitude, dx, dy]. These pseudo-images are patchified with the Qwen2.5-VL video grid and passed through the same frozen Qwen vision encoder to produce token-aligned motion features.

3. Additive connector

The trained visual_flow connector applies LayerNorm and a two-layer MLP to each motion token, then adds the result to the corresponding appearance token:

motion' = Linear(GELU(Linear(LayerNorm(motion))))
fused   = visual + motion'

The fused tokens replace the normal video-token embeddings before the language model forward pass. During inference, the LoRA adapter is merged into the base language model.

Component	Parameters	Training state	Included here
Qwen2.5-VL-7B base model and vision encoder	See base model	Frozen base weights; LoRA on LLM	No, downloaded separately
VideoFlow MOFNet	13,453,240	Frozen	Yes
`visual_flow` connector	14,692,864	Trained	Yes
LoRA adapter (`r=32`, `alpha=64`)	20,185,088	Trained	Yes
Trainable parameters	34,877,952	LoRA + connector	Yes
Bundled model parameters	48,331,192	Includes frozen MOFNet	Yes

Why a Controlled Experiment?

Current MLLMs show a consistent gap: they are substantially better at modeling space (objects, scenes, layouts) than self (motion, actions, agent state). To test whether explicit motion cues can alleviate this, we constructed SIS-Motion under strict controls:

✅ Same base model — Qwen2.5-VL-7B-Instruct
✅ Same training data — SIS-Motion-54K (perception + memory tasks only)
✅ Same LoRA configuration — r=32, α=64, identical target modules
✅ Same optimization — identical learning rate, schedule, and batch size
⬜ Only difference — the optical-flow-driven motion branch + additive connector

The visual-only SFT baseline serves as the direct comparison point. Any difference in results can be interpreted as the marginal contribution of motion-aware modeling.

Training Configuration

Setting	Value
Base model	`Qwen/Qwen2.5-VL-7B-Instruct`
Training data	SIS-Motion-54K (MCQ + Open QA + Description)
Data scope	Perception & memory tasks (no reasoning tasks)
LoRA rank `r`	32
LoRA alpha	64
LoRA dropout	0.05
LoRA targets	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Optimizer	AdamW
Precision	BF16
Trainable params	34,877,952 (20,185,088 LoRA + 14,692,864 connector)
Frozen params	Visual encoder, VideoFlow optical-flow estimator, LLM base (~7B)

Results

SIS-Bench

Model	Spatial Avg	Self Avg	Overall
ZeroShot (Qwen2.5-VL 3B)	65.8	40.5	53.6
SFT visual-only	72.0	60.3	66.4
SIS-Motion	74.2	63.7	69.1

Motion-aware modeling improves both spatial cognition (+2.2) and self-awareness (+3.4) over the visual-only SFT baseline. The gains are most pronounced on memory-level tasks, where temporal integration matters most.

Downstream Transfer — Zero-Shot UAV Navigation

Evaluated on the OpenUAV-QA navigation benchmark without any task-specific fine-tuning:

Model	Accuracy
Qwen2.5-VL 3B (backbone)	71.2%
SIS-Motion	92.2%

The +21.0 point gain demonstrates that motion-aware representations transfer beyond the original benchmark to practical UAV decision-making scenarios.

Note on reported results: The results above are from the paper's 3B backbone (Qwen2.5-VL-3B-Instruct). This repository hosts a 7B variant (Qwen2.5-VL-7B-Instruct) with the same architecture and training protocol. The 3B model can be reproduced using the open-source training code. We release the 7B version here as a stronger off-the-shelf checkpoint; the controlled-experiment conclusion (motion-aware gains over visual-only SFT) holds across both scales.

Repository Files

sis_motion_config.json       Architecture manifest (connector type, paths)
adapter_config.json          LoRA configuration (r=32, alpha=64)
adapter_model.safetensors    LoRA weights (20,185,088 params)
connector_weights.pt         Motion fusion MLP weights (14,692,864 params)
MOF_kitti.pth                Frozen VideoFlow MOFNet checkpoint (13,453,240 params)
images/poster.jpeg           Model poster

Note: The Qwen2.5-VL-7B-Instruct base weights are NOT included in this repository. They are downloaded automatically from HuggingFace at inference time.

Usage

This model package is not a standalone Transformers architecture. It must be loaded through the SIS-Motion evaluator, which constructs the motion branch before applying the PEFT adapter.

1. Clone the codebase

git clone https://github.com/IntelliSensing/Self-in-Space.git
cd Self-in-Space

2. Create the motion environment

With Conda:

bash scripts/setup_conda.sh motion
conda activate sis-motion-motion

Or with uv:

bash scripts/setup_uv.sh motion
source .venv-motion/bin/activate

The motion environment uses Python 3.10, PyTorch 2.6 with CUDA 12.4, Transformers 4.57.1, PEFT 0.18.1, and FlashAttention 2.7.4.

3. Prepare SIS-Bench

Place SIS-Bench under the registered data root or override the paths directly:

data/SIS-Bench/SIS-Bench.jsonl
data/SIS-Bench/frames/\

4. Run inference

The evaluator automatically downloads this model from HuggingFace, reads sis_motion_config.json, loads the VideoFlow optical-flow estimator (MOF_kitti.pth), applies the LoRA adapter, loads connector_weights.pt, and runs the full appearance-plus-motion pipeline:

MODEL_PATH=choucsan/SIS-Motion \
bash scripts/eval_motion.sh

An already downloaded model directory is also accepted:

MODEL_PATH=/path/to/SIS-Motion \
DATA_FILE=/path/to/SIS-Bench.jsonl \
FRAMES_DIR=/path/to/SIS-Bench/frames \
bash scripts/eval_motion.sh

The Qwen2.5-VL-7B base model is fetched from the base_model_name_or_path stored in adapter_config.json; it is not duplicated in this repository.

Note on the optical flow estimator: The default checkpoint (MOF_kitti.pth) is a VideoFlow MOFNet pretrained on KITTI. Replacing it with another optical-flow estimator (e.g., RAFT or GMFlow) requires adapting the optical-flow loading and output-conversion logic. The connector weights may not transfer to a different flow backbone and would likely require retraining.

Limitations

Not a SOTA model. SIS-Motion is a controlled research prototype built on a 7B backbone. Stronger base models would likely yield higher absolute scores under the same setup.
Optical flow estimation. The VideoFlow optical-flow estimator runs on each input video, adding compute not required by the visual-only baseline.
Training scope. SIS-Motion-54K covers perception and memory tasks. Reasoning-level improvements come from generalization, not direct supervision.
Inference cost. The runtime adds 28,146,104 module parameters over the base path (13,453,240 frozen MOFNet + 14,692,864 connector). The 20,185,088 LoRA parameters are merged into the base model before generation.

Citation

If you use SIS-Motion in your research, please cite:

@misc{zou2026sis,
      title={Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence},
      author={Zhishan Zou and Guoyan Sun and Zhiwei Wei and Jiancheng Pan and Yujie Li and Mugen Peng and Wenjia Xu},
      year={2026},
      eprint={2607.12477},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.12477},
}

Contact

For questions or collaboration requests:

[email protected]

Task: Video-Text-to-Text
Library: peft
License: apache-2.0
Created: Jul 12, 2026
Updated: Jul 18, 2026