DriveLMMo1

DriveLMM-o1: A Large Multimodal Model for Autonomous Driving Reasoning

Paper

DriveLMM-o1 is a fine-tuned large multimodal model designed for autonomous driving. Built on InternVL2.5-8B with LoRA-based adaptation, it leverages stitched multiview images to produce step-by-step reasoning. This structured approach enhances both final decision accuracy and interpretability in complex driving tasks like perception, prediction, and planning.

Key Features:

• Multimodal Integration: Combines multiview images for comprehensive scene understanding.
• Step-by-Step Reasoning: Produces detailed intermediate reasoning steps to explain decisions.
• Efficient Adaptation: Utilizes dynamic image patching and LoRA finetuning for high-resolution inputs with minimal extra parameters.
• Performance Gains: Achieves significant improvements in both final answer accuracy and overall reasoning scores compared to previous open-source models.

Performance Comparison:

Usage:

Load the model using the following code snippet:

from transformers import AutoModel, AutoTokenizer
import torch

path = 'ayeshaishaq/DriveLMMo1'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(
    path,
    trust_remote_code=True,
    use_fast=False
)

For detailed usage instructions and additional configurations, please refer to the OpenGVLab/InternVL2_5-8B repository.

Code: https://github.com/ayesha-ishaq/DriveLMM-o1

Limitations: While DriveLMM-o1 demonstrates strong performance in autonomous driving tasks, it is fine-tuned for domain-specific reasoning. Users may need to further fine-tune or adapt the model for different driving environments.