savi-finetuned-30slots

savi-finetuned-30slots

StoSAVi encoder from savi-pretrained-30slots, additionally fine-tuned on LMDrive with auxiliary heads (5 waypoints, traffic-light state, stop-sign state). Stage 1b of the object-centric LMDrive pipeline.

Overview

This is one of four model artefacts released alongside a thesis on replacing the multimodal sensor-fusion encoder in LMDrive with an object-centric encoder based on Stochastic Slot Attention from Video (SAVi), evaluated on the CARLA LangAuto benchmark. The four artefacts are:

• ni-eminen/savi-pretrained-30slots -- Stage 1a encoder
• ni-eminen/savi-finetuned-30slots -- Stage 1b encoder
• ni-eminen/object-centric-lmdrive-4qt -- Q-Former + LLM (4 query tokens)
• ni-eminen/object-centric-lmdrive-8qt -- Q-Former + LLM (8 query tokens)

Architecture

• architecture: StoSAVi + waypoint / traffic-light / stop-sign heads
• num slots: 30
• slot layout: 24 general + 1 traffic-light/stop-sign role + 5 waypoint role
• slot size: 256
• input resolution: [192, 192]
• clip length: 6
• kld method: var-0.01
• waypoints head: GRU predictor over 5 role slots -> 5x(dx,dy)
• traffic light head: Linear(256, 2)
• stop sign head: Linear(256, 2)

Inputs

Same as savi-pretrained-30slots: 192x192 RGB BEV, [0, 1] normalisation, temporal clips of 6 frames. Additionally accepts a scalar velocity for velocity-conditioning of the role slot initial latents.

Training

Fine-tuned from savi-pretrained-30slots on LMDrive using an L1 waypoint loss (weight 0.5), CE traffic-light loss (0.1), CE stop-sign loss (0.01), and MSE reconstruction loss (1.0). Optimizer AdamW with initial LR 7.5e-4, cosine schedule, weight decay 0.05, gradient clipping 5. Batch size 6, 4 A100-40GB GPUs, 2 epochs (approximately 35 hours).

Evaluation

Evaluated on the same 72 held-out BEV frames used for savi-pretrained-30slots.

Fine-tuning increases whole-image MSE by 41% and foreground MSE by 107% relative to the pretrained encoder while leaving object-partition quality (FG-ARI, mIoU) essentially unchanged.

Citation

@mastersthesis{nieminen2026objcentric,
  title  = {Object-centric world models for language-instructed autonomous driving},
  author = {Nieminen, Matias},
  year   = {2026},
  school = {Aalto University}
}

Related repositories

• Encoder training code: https://github.com/ni-eminen/SAVi-for-LMDrive-BEV
• Instruction-fine-tuning + evaluation code: https://github.com/ni-eminen/lmdrive-fix