Downloads · 30 days
26
16% of all-time downloads
zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA
Qwen2-VL-7B-DriveLM-LoRA is a image-text-to-text model from zhudanburujiandan. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
A LoRA adapter that fine-tunes Qwen/Qwen2-VL-7B-Instruct on the DriveLM-nuScenes v1.1 graph visual question-answering dataset. Turns the base Qwen2-VL into a driving perception + prediction + planning assistant that r…
Downloads · 30 days
26
16% of all-time downloads
All-time downloads
162
Public
Repo size
334 MB
Likes
0
Public
Click a slice to open those files.
.safetensors323 MB · 97%
From the Hugging Face model README
A LoRA adapter that fine-tunes Qwen/Qwen2-VL-7B-Instruct on the DriveLM-nuScenes v1.1 graph visual question-answering dataset. Turns the base Qwen2-VL into a driving perception + prediction + planning assistant that reads a front-view dashcam image and answers structured driving questions in the DriveLM convention (Perception / Prediction / Planning / Behavior).
| Metric on 100 held-out val QAs (scene-based 90/10 split, seed=42) | Base Qwen2-VL-7B (zero-shot) | This adapter | Δ |
|---|---|---|---|
| Exact match | 9 / 100 | 62 / 100 | +53 |
| Fuzzy substring match | 11 / 100 | 65 / 100 | +54 |
| Verbose (pred > 3× GT length) | 14 / 100 | 0 / 100 | -14 |
Per DriveLM level (25 QAs each):
| Level | Zero-shot exact | Adapter exact | Δ |
|---|---|---|---|
| Perception | 5 | 16 | +11 |
| Prediction | 4 | 22 | +18 |
| Planning | 0 | 13 | +13 |
| Behavior | 0 | 11 | +11 |
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
import torch
BASE = "Qwen/Qwen2-VL-7B-Instruct"
ADAPTER = "zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA"
model = Qwen2VLForConditionalGeneration.from_pretrained(
BASE, dtype=torch.bfloat16, device_map="cuda",
attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()
processor = AutoProcessor.from_pretrained(BASE,
min_pixels=100*28*28, max_pixels=512*28*28)
messages = [
{"role": "system",
"content": "You are a driving perception assistant. You see the front "
"camera view of the ego vehicle. Answer the question concisely "
"based only on what is visible."},
{"role": "user", "content": [
{"type": "image", "image": "path/to/front_cam.jpg"},
{"type": "text", "text": "Predict the behavior of the ego vehicle."},
]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs,
padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:],
skip_special_tokens=True)[0])
Expected style of output on driving images:
"There are two cars and one truck to the front of the ego car.""Yes." / "The ego vehicle.""Keep going at the same speed.""The ego vehicle is going straight. The ego vehicle is driving slowly."seed=42.LoraConfig(
r=32, lora_alpha=64, lora_dropout=0.05, bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj","k_proj","v_proj","o_proj",
"gate_proj","up_proj","down_proj"],
)
Trainable parameters: 80,740,352 / 8,372,115,968 ≈ 0.96 %.
per_device_batch_size=1, gradient_accumulation_steps=8).optim="adamw_8bit" (bitsandbytes) + bf16=True + gradient_checkpointing=True.lr=2e-5, warmup_ratio=0.03, weight_decay=0.max_pixels=512*28*28 (~480 vision tokens per image), max_length=1024.attn_implementation="sdpa".3.6 s/step.train_loss ≈ 0.30 (epoch avg), eval_loss ≈ 0.18 (plateau by 52 % of epoch).| Train QAs | Wall time | Val exact | Val fuzzy |
|---|---|---|---|
| 500 (smoke, 3.2 ep) | 12 min | 40 | 40 |
| 20,000 (1 ep) | 2 h 33 min | 56 | 59 |
| 154,363 (1 ep, this release) | 19 h 33 min | 62 | 65 |
Marginal cost/benefit degrades sharply past 20 k: 7.7× more compute buys +6 exact points. If you re-train on new driving data, 20 k QAs is likely the sweet spot for the first pass.
"driving fast / slowly / normally" predictions in the Behavior category rely on static visual cues (traffic density, road curvature). Behavior exact match tops out around 44 % (11 / 25 in the eval). Real temporal input (multi-frame from nuScenes) is expected to lift this further."to the back" / "back-left/right" / "behind" will hallucinate. A 3-camera panorama variant was attempted and did not help (dropped exact from 62 → 52 due to per-camera resolution loss)."Yes." / "The ego vehicle." / "Please proceed."). It may under-generate on other datasets that expect longer, unstructured answers.Training pipeline: collator with token-search label masking (<|im_start|>assistant\n boundary is found in tokenized input_ids — more robust than the naive apply_chat_template prompt-length approach which suffers from BPE boundary drift).
Eval harness: 100 QAs sampled balanced across 4 DriveLM levels (25 each) from the 70 val scenes, seed=42. Metrics computed on lowercase-stripped strings.
If you use this adapter, cite the underlying works:
@article{qwen2vl,
title={{Qwen2-VL}: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
author={Wang, Peng and Bai, Shuai and others},
journal={arXiv preprint arXiv:2409.12191},
year={2024},
}
@inproceedings{drivelm,
title={{DriveLM}: Driving with Graph Visual Question Answering},
author={Sima, Chonghao and Renz, Katrin and others},
booktitle={ECCV},
year={2024},
}
Apache-2.0 (inherits from Qwen2-VL-7B-Instruct and DriveLM-nuScenes v1.1).