Qwen-Drive-Kinematic-Planner

Qwen-Drive Kinematic Planner (v11)

An end-to-end Vision-Language-Action (VLA) trajectory planning model for autonomous driving, fine-tuned with LoRA on top of the Qwen-Drive-1.0-4B VLM backbone, combined with a Differentiable Kinematic Vehicle Planner, trained on the nuScenes dataset.

Instead of unconstrained coordinate regression (which often causes spatial teleportation and dynamically infeasible maneuvers), this model predicts physically constrained longitudinal accelerations a_t and yaw rates ω_t over a 5.0-second horizon (50 steps at Δt = 0.1s). Trajectories (x, y, ψ) are produced via a differentiable kinematic (unicycle) integration of these signals.

📌 Model Predictions & Outputs Summary

For every frame, given 3 consecutive front-camera images and current CAN telemetry ([speed_kmh, accel_norm]), UnifiedE2EModel v11 outputs 5 distinct variables:

1. Direct Vehicle Control Commands (via controller head)

• 🏎️ Target Longitudinal Speed: Denormalized to km/h. Direct target speed for throttle/brake actuators.
• 🛞 Steering Wheel Angle: Denormalized to radians / degrees. Direct command for the steering rack (MAE: ~0.95°).

2. Internal Kinematic Motion Parameters (via kinematic_planner head)

• 🚀 Longitudinal Acceleration Profile (a_1 to a_50): Predicted for 50 timesteps, constrained to [-6.0, +4.0] m/s² (asymmetric braking/acceleration limits).
• 🔄 Yaw Rate Profile (w_1 to w_50): Predicted angular velocity for 50 timesteps, constrained to [-1.5, +1.5] rad/s.

3. Integrated Physical 5.0-Second Trajectory

• 📍 Future Ego Waypoints (x, y, yaw): Calculated for 50 timesteps (5.0s horizon at Δt = 0.1s) by mathematically integrating acceleration and yaw rate from initial CAN speed v0. Guaranteed to be smooth and kinematics-compliant without spatial teleportation.

⚠️ Note on the Base Model

This checkpoint is a LoRA fine-tune of the VLM component inside Qwen/Qwen-Drive-1.0-4B, loaded via the official qwen_drive package (QwenDriveForPlanning.from_pretrained(...))

The original Planning Expert (flow matching) and BEV perception head from Qwen-Drive-1.0 are not used — this repository replaces them with a custom differentiable kinematic planning head, trained from scratch on top of the frozen (+LoRA) VLM backbone.

🏆 Key Benchmarks & Results (nuScenes v1.0-trainval, validation split)

Baseline definition: the same VLM backbone and training pipeline, but with the kinematic planner replaced by direct MLP regression of (x, y) coordinates — i.e. this is an internal ablation, not a comparison against a third-party method.

Metric protocol note: ADE is averaged over the full 0–5.0s horizon (all 50 steps), not only the final timestep. FDE is the displacement error at t=5.0s only. These are not directly comparable to ADE@5s figures reported by other systems that measure only the endpoint.

📐 Architecture

🛠️ Unified Trajectory & Control Co-Design

Rather than relying solely on coordinate outputs, the model simultaneously generates a dynamically feasible 5.0-second trajectory via a differentiable kinematic integrator and outputs direct, calibrated CAN-bus actuation commands—specifically, target longitudinal speed and steering wheel angle (δ).

🧬 Differentiable Kinematic Integration

$$v_t = \max\left(0,\ v_0 + \sum_{\tau=1}^{t} a_\tau \Delta t\right)$$

$$\psi_t = \sum_{\tau=1}^{t} \omega_\tau \Delta t$$

$$x_t = \sum_{\tau=1}^{t} v_\tau \cos(\psi_\tau), \Delta t, \qquad y_t = \sum_{\tau=1}^{t} v_\tau \sin(\psi_\tau), \Delta t$$

Acceleration and yaw rate are constrained via tanh scaling:

• a_t ∈ [-6.0, +4.0] m/s² (asymmetric: braking limit is higher than acceleration limit)
• ω_t ∈ [-1.5, +1.5] rad/s

All operations are differentiable, so gradients from the trajectory loss flow back through the integrator into the predicted controls and, further, into the visual and CAN features — no separate supervision on a_t, ω_t is required.

⚙️ Other Design Choices

• Temporal context: 3 consecutive CAM_FRONT frames (t-2, t-1, t).
• Timeline synchronization: trajectories are built from sensor-rate sample_data timestamps (~12 Hz) rather than the 2 Hz keyframe grid, then linearly interpolated to a strict Δt = 0.1s grid.
• Trajectory validity filtering: samples are discarded if less than 80% of the 5.0s horizon can be covered by available future poses, and if the required historical frames are not present on disk.
• Jerk regularization: an additional loss term penalizes the third time derivative of position to encourage smooth trajectories.
• Time-weighted trajectory loss: per-step loss weights increase linearly from 0.8 (t=0.1s) to 2.0 (t=5.0s), penalizing late-horizon errors more.

<img src="simulation.gif">

⚙️ Training Details

📦 Checkpoint Contents

best_autopilot_v11.pt (~24 MB) contains:

• lora_state_dict: LoRA adapters for the Qwen-Drive-1.0 VLM attention projections (q_proj, k_proj, v_proj, o_proj).
• head_state_dict: weights for scene_proj, cross_attn, cross_norm, dyn_embed, kinematic_planner, controller (see model.py).
• stats: normalization parameters (speed_mean, speed_std, steer_mean, steer_std) computed on the training split.

🚗 Usage & Inference

⚠️ Important: the question string must exactly match the prompt used during training: "Predict future trajectory and vehicle control signals." The model was never trained with a different or empty prompt, and changing it will shift the VLM's hidden states outside the distribution the planning/control heads were fitted to.

Requirements

git clone https://github.com/QwenLM/Qwen-Drive-1.0 qwen-drive
cd qwen-drive && pip install -e . --no-build-isolation
hf download Qwen/Qwen-Drive-1.0-4B --local-dir Qwen-Drive-1.0-4B
pip install torch peft huggingface_hub pillow

import sys
import torch
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
from peft import LoraConfig, get_peft_model, set_peft_model_state_dict

sys.path.append("./qwen-drive/src")
from qwen_drive import QwenDriveConfig, QwenDriveProcessor, QwenDriveForPlanning

REPO_ID = "Aleton/Qwen-Drive-Kinematic-Planner"
QWEN_DRIVE_PATH = "./Qwen-Drive-1.0-4B"
device = "cuda" if torch.cuda.is_available() else "cpu"

hf_hub_download(repo_id=REPO_ID, filename="model.py", local_dir=".")
ckpt_path = hf_hub_download(repo_id=REPO_ID, filename="best_autopilot_v11.pt")
ckpt = torch.load(ckpt_path, map_location=device)
stats = ckpt["stats"]

from model import UnifiedE2EModel

base_model = QwenDriveForPlanning.from_pretrained(
    QWEN_DRIVE_PATH, dtype=torch.bfloat16, attn_implementation="sdpa",
)
vlm_core = getattr(base_model, "vlm", getattr(base_model, "model", base_model))

tokenizer = AutoTokenizer.from_pretrained(QWEN_DRIVE_PATH)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

config_qwen = QwenDriveConfig.from_pretrained(QWEN_DRIVE_PATH)
processor = QwenDriveProcessor(tokenizer, config_qwen)

lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
vlm_lora = get_peft_model(vlm_core, lora_config)
set_peft_model_state_dict(vlm_lora, ckpt["lora_state_dict"])

model = UnifiedE2EModel(vlm_lora, hidden_dim=512, traj_points=50, traj_dt=0.1).to(device)
model.load_state_dict(ckpt["head_state_dict"], strict=False)
model.eval()

encoded = processor.encode_vqa(
    images=[...],  # 3 PIL.Image frames: [t-2, t-1, t]
    question="Predict future trajectory and vehicle control signals.",
)
inputs = {k: v.to(device) for k, v in encoded.items()}
inputs["attention_mask"] = torch.ones_like(inputs["input_ids"])
inputs["mm_token_type_ids"] = (inputs["input_ids"] == processor.image_token_id).long()
inputs["can_state"] = torch.tensor([[36.0, 0.0]], dtype=torch.float32, device=device)

with torch.no_grad():
    outputs = model(**inputs)

trajectory = outputs["trajectory"][0].cpu().numpy()
controls = outputs["controls"][0].cpu().numpy()
target_speed_kmh = controls[0] * stats["speed_std"] + stats["speed_mean"]
steer_angle_rad = controls[1] * stats["steer_std"] + stats["steer_mean"]

print(f"Target Speed : {target_speed_kmh:.1f} km/h")
print(f"Steering     : {torch.rad2deg(torch.tensor(steer_angle_rad)):+.2f}°")

🎯 Intended Use

This is a research prototype for studying kinematically-constrained trajectory planning in VLA models. It is trained and evaluated only on nuScenes front-camera data.

Out of scope: real-world deployment, use as a safety-critical driving system, or use on sensor configurations / geographies not represented in nuScenes. Predictions have not been validated in closed-loop or real-vehicle settings.

⚠️ Limitations

• Single Camera Field-of-View: Uses only the front camera (CAM_FRONT), leaving side and rear blind spots unmonitored.
• Simplified Dynamic Model: Uses a unicycle kinematic model — no lateral tire slip, roll/pitch dynamics, or road-surface interaction are modeled.
• Telemetry Dependency: Relies on accurate CAN telemetry (v0) as the integrator's initial condition; degraded telemetry will bias the entire predicted trajectory.
• Dataset Generalization: Evaluated only on nuScenes v1.0-trainval; generalization to other sensor setups, countries, or weather conditions is untested.
• Label Noise: ADE/FDE are computed against nuScenes ego-pose ground truth interpolated from ~12 Hz keyframes; label noise in ego-pose is not separately quantified.

📄 License & Terms of Use

• Code & Architecture: Released under the Apache-2.0 License.
• Model Weights (best_autopilot_v11.pt): Restricted to Non-Commercial / Research Use Only, inheriting the license terms of the nuScenes Dataset (CC BY-NC-SA 4.0).

📜 Citation

@misc{aleton2026qwendrivev11,
  title={Qwen-Drive Kinematic Planner (v11): Differentiable Kinematic Trajectory Planning on nuScenes},
  author={Vishnevskiy, Aleksey},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/Aleton/Qwen-Drive-Kinematic-Planner}}
}

@misc{zhou2026qwendrive10,
  title={Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving},
  author={Xin Zhou and Zongchuang Zhao and Zhibo Yang and Mingsheng Li and Humen Zhong and
          Shuai Bai and Du Chu and Ruizhe Chen and Zhaohai Li and Jun Tang and Qiuyue Wang and
          Mingkun Yang and Jiazhao Zhang and Dayiheng Liu and Dingkang Liang and Xiang Bai},
  year={2026},
  eprint={2609.00111},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2609.00111}
}

@article{nuscenes2019,
  title={nuScenes: A multimodal dataset for autonomous driving},
  author={Holger Caesar and Varun Bankiti and Alex H. Lang and Sourabh Vora and
          Venice Erin Liong and Qiang Xu and Anush Krishnan and Yu Pan and
          Giancarlo Baldan and Oscar Beijbom},
  journal={arXiv preprint arXiv:1903.11027},
  year={2019}
}