mcvo

MCVO — Multi-frame, Camera-only Visual Odometry

A transformer that predicts relative camera pose and camera intrinsics from images alone, trained self-supervised on ~80k frames of unlabeled video. Pretrained depth (UniDepth), optical-flow (UniMatch) and calibration (AnyCalib) networks supervise a flow re-projection loss at training time only; none of them runs at inference. Research code from TU Munich (Chair of Computer Vision, Prof. Cremers), in development; code and benchmark: https://github.com/kalman17/mcvo

Files

• mcvo_e3p_calib.pt — recommended: pose + intrinsics (E3′ weights with the calibration head distilled on them; pose output identical to mcvo_e3p.pt)
• mcvo_e3p.pt — the same weights without the trained calibration head (pose only)
• mcvo_e3.pt, mcvo_e3_calib.pt — the previous release, trained with a depth-convention error in the loss (see the changelog in the repository); kept for reference

import torch
from mcvo.model import MCVO                     # from the GitHub repo, PYTHONPATH=.

ck = torch.load("mcvo_e3p_calib.pt", map_location="cpu", weights_only=False)
a = ck["args"]
model = MCVO(backbone=a["backbone"], d_model=a["d_model"], depth=a["depth"], heads=a["heads"]).eval()
model.load_state_dict(ck["model_state_dict"])
out = model(images=frames)     # frames: [1, N, 3, 336, 336], RGB in [0, 1]
out["poses"]                   # [1, N, 1, 4, 4]  cam_i -> cam_{i+1}, last = identity
out["calib"]                   # [1, N, 4]        fx, fy, cx, cy (px)

Model

• Frozen DINOv2-base backbone (86 M) → patch tokens per frame, plus one learned camera token per frame.
• 10 transformer blocks, each: temporal attention across frames per patch position → spatial attention within each frame (camera token included) → MLP. d = 640, 8 heads. 67 M trained parameters, 154 M total.
• Heads: pose (MLP on adjacent camera tokens → translation + quaternion), calibration (linear on the camera token → focal length, principal point; distilled from AnyCalib, head-only), uncertainty (linear per patch; used only in the loss).
• Training: 6 epochs on 8-frame clips (RealEstate10K, YouTube-VOS, EpicKitchens, WalkingTours preprocessed with the AnyCam pipeline), batch 4, lr 1.5e-4, uncertainty-weighted flow re-projection loss, one A40-class GPU.
• Input: N frames, 336×336, RGB in [0, 1]. Output: N−1 relative poses and per-frame intrinsics.

Numbers (NVIDIA A40, 4-frame windows, medians; same harness for every model)

Similar rotation accuracy to AnyCam at 5.5× lower latency and 5× less memory; roughly twice the rotation error of the billion-parameter supervised models at 8–14× less memory and 2–8× lower latency. Translation direction is good on driving video and weak on small-baseline indoor video (Sintel 104°, TUM-RGBD 75°; 90° is chance). Trajectory-level Sim3 ATE from chained 8-frame windows without bundle adjustment: Sintel 0.29, TUM-RGBD 0.14 (AnyCam feed-forward: 0.10 on Sintel). Full table, protocol, raw rows and the dated changelog: GitHub repo.

Attribution

Builds on AnyCam (Wimbauer et al., CVPR 2025) for the self-supervised loss and data pipeline, DINOv2 (Meta) as backbone, UniDepth, UniMatch and AnyCalib as training-time teachers. Architecture follows the time-space attention design of FVO (Yugay et al., 2025), trained self-supervised instead of with ground-truth poses. MIT.