object-centric-lmdrive-4qt

object-centric-lmdrive-4qt

Object-centric LMDrive with 4 Q-Former query tokens. LLaVA-1.5-7B conditioned on 30 SAVi slots via a Q-Former; produces future waypoints and end-of-instruction flags for the LangAuto benchmark.

Overview

This is one of four model artefacts released alongside a thesis on replacing the multimodal sensor-fusion encoder in LMDrive with an object-centric encoder based on Stochastic Slot Attention from Video (SAVi), evaluated on the CARLA LangAuto benchmark. The four artefacts are:

• ni-eminen/savi-pretrained-30slots -- Stage 1a encoder
• ni-eminen/savi-finetuned-30slots -- Stage 1b encoder
• ni-eminen/object-centric-lmdrive-4qt -- Q-Former + LLM (4 query tokens)
• ni-eminen/object-centric-lmdrive-8qt -- Q-Former + LLM (8 query tokens)

Architecture

• encoder: StoSAVi (30 slots, 256-dim)
• adapter: BLIP-2 Q-Former (BERT-base-uncased backbone, 4 query tokens)
• llm: LLaVA-1.5-7B (Vicuna-7B backbone, LoRA fine-tuned)
• waypoint head: 2-layer MLP (4096 -> 4096 -> 10)
• end head: 2-layer MLP (4096 -> 4096 -> 2)
• input frames per instruction: 40
• sample interval: 2
• max txt len: 64

Inputs

An instruction window of up to 40 BEV frames (at 10Hz effective after sub-sampling from CARLA's 20Hz simulator) plus a natural language instruction (max 64 tokens after tokenization).

Training

Trained on the full LMDrive corpus (~2TB) for 15 epochs (approximately 82 hours across 4 A100-40GB GPUs). Optimizer AdamW (betas=(0.9, 0.999)), initial LR 1e-4 with linear warmup over 2000 steps then cosine decay to 1e-5, weight decay 0.06, batch size 4 per GPU. Encoder and LLM frozen; Q-Former + adapters

• prediction MLPs trained.

Evaluation

Evaluated on the LangAuto benchmark (32 long routes from CARLA), averaged across 3 repetitions.

Citation

@mastersthesis{nieminen2026objcentric,
  title  = {Object-centric world models for language-instructed autonomous driving},
  author = {Nieminen, Matias},
  year   = {2026},
  school = {Aalto University}
}

Related repositories

• Encoder training code: https://github.com/ni-eminen/SAVi-for-LMDrive-BEV
• Instruction-fine-tuning + evaluation code: https://github.com/ni-eminen/lmdrive-fix