online_rnn_transformer_video_prediction

Comparing transformers and online-trained RNNs for video prediction using the PCA respiratory motion model.

This page documents the reference implementation used in the following paper on chest and liver cine-MRI frame forecasting with transformers and online training of vanilla RNNs.

Paper

• Title: Frame forecasting in cine MRI using the PCA respiratory motion model: comparing recurrent neural networks trained online and transformers
• Authors: Michel Pohl, Mitsuru Uesaka, Hiroyuki Takahashi, Kazuyuki Demachi, Ritu Bhusal Chhatkuli
• arXiv: https://doi.org/10.48550/arXiv.2410.05882
• Published version: https://doi.org/10.1016/j.compmedimag.2026.102755 -
• Hugging Face paper page: https://huggingface.co/papers/2410.05882

Code

• GitHub repository: https://github.com/pohl-michel/2D-MR-image-prediction
• Latest release (Zenodo): https://doi.org/10.5281/zenodo.13896201

Description

This work compares linear filters, sequence-specific and population transformer encoders and online learning algorithms for RNNs (real-time recurrent learning [RTRL], unbiased online recurrent optimization, decoupled neural interfaces, and sparse one-step approximation [SnAp-1]) to forecast thoraco-abdominal cine-MR image sequences. PCA decomposes the motion in the input video, estimated using the Lucas-Kanade optical-flow algorithm, into principal components and time-dependent weights. The latter are predicted using the algorithms above to estimate future motion, and in turn, the future frame, by warping the initial image. RTRL and SnAp- 1 outperformed the other algorithms at medium-to-long horizons. Sequence-specific transformers were competitive for low-to-medium horizons, but transformers remained overall constrained by data scarcity and domain shift between datasets. Predicted frames visually resembled the ground truth, with notable errors occurring near the diaphragm at end-inspiration and regions affected by out-of-plane motion.

Citation

@article{pohl2026frame,
  title={Frame forecasting in cine MRI using the PCA respiratory motion model: comparing recurrent neural networks trained online and transformers},
  author={Pohl, Michel and Uesaka, Mitsuru and Takahashi, Hiroyuki and Demachi, Kazuyuki and Chhatkuli, Ritu Bhusal},
  journal={Computerized Medical Imaging and Graphics},
  pages={102755},
  year={2026},
  publisher={Elsevier}
}