Qwen3-VL-2B-Bench2Drive-Perception

Qwen3-VL-2B — Bench2Drive Front-View Perception

A LoRA adaptation of Qwen3-VL-2B-Instruct for front-camera perception on Bench2Drive. It answers metric, planner-consumable questions about the scene ahead: where each vehicle is, how far the lane is clear, what the traffic light governing your lane shows, whether the gap to the lead vehicle is closing.

Merged and standalone — load with from_pretrained, no adapter needed.

Input format

Front camera only: the current frame plus three history frames at 0.5 s intervals.

Front-view frames (oldest to newest):
[t-1.5s]<image>[t-1.0s]<image>[t-0.5s]<image>[t=0 current]<image>
<question>

Ego-centric coordinates: origin (0,0), X = lateral (right +), Y = longitudinal (forward +), metres. Trained at a vision budget of MAX_PIXELS=131072 (≈512×256); set the same at inference.

What it was trained on

174,681 samples, 68,187 distinct image tuples, 1,480 Bench2Drive clips.

Labels are derived from the simulator ground truth in Bench2Drive-Full-Sup (anno/*.json.gz): 3D boxes are projected through the CAM_FRONT intrinsics and extrinsics, and only objects that actually resolve in the front view are used. Every task's answer distribution is balanced by construction, so the language prior cannot carry it.

13 tasks: object counting, nearest-object localisation, full-neighbourhood localisation, free distance in own lane, lead-vehicle type/distance, gap dynamics, traffic-light state, category presence, side occupancy, lead-vehicle motion, ego speed state, ego steering state, and 3 s ego trajectory.

Evaluation

6,000 held-out items from 120 clips disjoint from training. Each task is reported against that set's own majority-class baseline, and against a blind control — the identical prompt with every image replaced by a uniform grey frame. The sighted-minus-blind margin is the visual contribution; absolute accuracy alone is not evidence of perception.

Localisation uses a 3 m tolerance. The multi-object row is all-or-nothing — every gold object matched by a distinct prediction within 3 m and the same object count. Per object, which is what a planner actually consumes:

Blinded, it matches zero objects — the cleanest visual evidence in the suite.

Caveat on trajectory. Blind accuracy is 37.5%, far above the 1.8% majority baseline: a stopped ego yields all-zero waypoints and steady cruising yields a predictable straight line, both guessable without the image. Only ~15 points of that task's 52.8% are visual. Do not cite trajectory accuracy as perception evidence. Every other task's blind score sits on its majority baseline.

Against the prior checkpoint

Same eval set, same protocol:

Training

LoRA r=64, α=128, all linear layers of the language model plus the vision-language aligner; ViT frozen. bf16, lr 1e-4 cosine, warmup 0.03, weight decay 0.1, global batch 120, max length 6144, 2 epochs, 3×A100-40GB, ms-swift.

Freezing the ViT is deliberate: a matched ablation that unfroze it with a 10×-lower ViT learning rate (1e-5) scored 81.3% vs 81.1% on the comparable task subset — no benefit for 25M extra parameters.

Limitations

• Front camera only; objects outside the front view are invisible to it, and questions about them are unanswerable rather than merely hard.
• Trained and evaluated in CARLA/Bench2Drive. No real-world validation.
• Trajectory output is largely prior-driven (see caveat above).
• Traffic-light colour is learned from the simulator's affects_ego flag; frames where a light governs the lane but is not visible were excluded from training rather than labelled "none".
• Pedestrians, trucks and buses appear in under 1% of frames, so the model has effectively no supervision for them.