Downloads · 30 days
8
13% of all-time downloads
zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA-v3
Qwen2-VL-7B-DriveLM-LoRA-v3 is a image-text-to-text model from zhudanburujiandan. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
A LoRA adapter that fine-tunes Qwen/Qwen2-VL-7B-Instruct on DriveLM-nuScenes v1.1 while giving the model a numeric ego-speed hint at inference time. Compared with the previous v1 (visual-only) LoRA, v3 breaks the long…
Downloads · 30 days
8
13% of all-time downloads
All-time downloads
60
Public
Repo size
334 MB
Likes
0
Public
Click a slice to open those files.
.safetensors323 MB · 97%
From the Hugging Face model README
A LoRA adapter that fine-tunes Qwen/Qwen2-VL-7B-Instruct on DriveLM-nuScenes v1.1 while giving the model a numeric ego-speed hint at inference time. Compared with the previous v1 (visual-only) LoRA, v3 breaks the long-standing behavior-accuracy ceiling on this dataset (44% → 60%+).
| Metric (100 held-out val QAs, seed=42) | Zero-shot | v1 (visual only, 154k train) | v3 (visual + speed, 154k train) | Δ |
|---|---|---|---|---|
| Exact match | 9 | 62 | 66 | +4 |
| Fuzzy substring | 11 | 65 | 70 | +5 |
| Verbose (pred > 3× GT) | 14 | 0 | 0 | 0 |
Per DriveLM level (25 QAs each):
| Level | v1 exact | v3 exact | Δ |
|---|---|---|---|
| Perception | 16 | 16 | 0 |
| Prediction | 22 | 22 | 0 |
| Planning | 13 | 13 | 0 |
| Behavior | 11 (44%) | 15 (60%) | +4 (+16pp) |
All gains land on the behavior level — v3 matches v1 on perception / prediction / planning and adds +4 exact points where the label class is directly speed-derived ("driving fast / normal / slowly / not moving").
A companion v3-20k variant (trained on 20k QAs instead of 154k) trades total exact for even stronger behavior — see the v3-20k branch — 17/25 behavior (68%) at 64 total exact.
One thing only: at inference time, prepend "Ego speed: X.Y m/s.\n" to the user prompt (X.Y = ego vehicle's speed magnitude in m/s). Everything else is identical to v1 — same LoRA rank, same base, same DriveLM QA schema.
Where to get the speed value:
ego_pose.translation between consecutive samples (script below).Why it works: behavior labels in DriveLM are a discretization of ego speed. Numeric grounding lets the LLM directly threshold-classify instead of guessing from static visual cues. See W17 negative log for the ablation that ruled out visual-temporal approaches.
Same as v1 (front-view dashcam → DriveLM-style Q&A), plus:
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
import torch
BASE = "Qwen/Qwen2-VL-7B-Instruct"
ADAPTER = "zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA-v3"
model = Qwen2VLForConditionalGeneration.from_pretrained(
BASE, dtype=torch.bfloat16, device_map="cuda", attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()
processor = AutoProcessor.from_pretrained(BASE, min_pixels=100*28*28, max_pixels=512*28*28)
ego_speed_m_s = 5.5 # from odometry / pose diff
question = "Predict the behavior of the ego vehicle."
user_text = f"Ego speed: {ego_speed_m_s:.1f} m/s.\n{question}"
messages = [
{"role": "system",
"content": "You are a driving perception assistant. You see the front camera view of "
"the ego vehicle. Answer the question concisely based only on what is visible. "
"You will also be told the current ego speed in m/s at the start of each prompt."},
{"role": "user", "content": [
{"type": "image", "image": "path/to/front_cam.jpg"},
{"type": "text", "text": user_text},
]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
Expected outputs (behavior questions):
"The ego vehicle is going straight. The ego vehicle is not moving.""The ego vehicle is going straight. The ego vehicle is driving slowly.""The ego vehicle is going straight. The ego vehicle is driving with normal speed.""The ego vehicle is going straight. The ego vehicle is driving fast.""The ego vehicle is going straight. The ego vehicle is driving very fast."Other levels (perception / prediction / planning) still work; the speed prefix is harmless (and slightly helpful) there.
If you're evaluating on nuScenes / DriveLM directly, differentiate ego pose:
from nuscenes.nuscenes import NuScenes
import numpy as np
nusc = NuScenes(version="v1.0-trainval", dataroot="path/to/nuscenes", verbose=False)
def sample_speed_m_s(sample_token: str) -> float:
s = nusc.get('sample', sample_token)
scene_samples = [x for x in nusc.sample if x['scene_token'] == s['scene_token']]
scene_samples.sort(key=lambda x: x['timestamp'])
idx = next(i for i, x in enumerate(scene_samples) if x['token'] == sample_token)
lo = max(0, idx - 1); hi = min(len(scene_samples) - 1, idx + 1)
def pose(x):
sd = nusc.get('sample_data', x['data']['CAM_FRONT'])
return np.array(nusc.get('ego_pose', sd['ego_pose_token'])['translation']), sd['timestamp']
pa, ta = pose(scene_samples[lo]); pb, tb = pose(scene_samples[hi])
dt = (tb - ta) / 1e6
return float(np.linalg.norm(pb - pa) / dt) if dt > 0 else 0.0
Central difference (previous + next samples) gives ~0.5-1s smoothing that matches training.
On DriveLM's 3,652 behavior QAs (train split), the "driving fast / normal / slowly / not moving" categories map cleanly onto pose-diff speed:
| Label | Median speed | p10 – p90 |
|---|---|---|
| not moving | 0.00 m/s | 0.00 – 1.14 |
| slow | 3.11 m/s | 0.68 – 4.85 |
| normal | 5.22 m/s | 3.69 – 6.59 |
| fast | 7.29 m/s | 5.77 – 8.81 |
The p10/p90 bands barely overlap, so thresholds around [1.5, 4.0, 6.0] m/s already recover the labels almost perfectly. What v3 learns is essentially that mapping (with a small residual for tie-breaking via visual scene).
Same as v1, except:
"Ego speed: X.Y m/s.\n" (computed from nuScenes ego_pose diffs).eval_loss plateau 0.175 (v1 was 0.19; small but real gain in loss too).Compute delta vs v1: same ~20h. Speed prefix costs ~15 extra tokens per sample; no measurable slowdown.
Takeaway: for VLMs to classify a physical quantity that has a native numeric label (speed, distance, count), feed the number. Denser video, more cameras, and longer training all failed on this axis; a single float in the prompt lifted 44% → 60%.
Same as v1 (see v1 README) plus:
Same base + dataset citations as v1 (see v1 README):
@article{qwen2vl,
title={{Qwen2-VL}: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
author={Wang, Peng and Bai, Shuai and others},
journal={arXiv preprint arXiv:2409.12191},
year={2024},
}
@inproceedings{drivelm,
title={{DriveLM}: Driving with Graph Visual Question Answering},
author={Sima, Chonghao and Renz, Katrin and others},
booktitle={ECCV},
year={2024},
}
Apache-2.0 (inherits from Qwen2-VL-7B-Instruct and DriveLM-nuScenes v1.1).