ur5_qwen25vl_act_chunk30

Ngseo/ur5_qwen25vl_act_chunk30

Qwen2.5-VL-Act (Option-B) policy fine-tuned on Ngseo/ur5_teleop_multitask (UR5 real-robot teleop, 350 episodes, 7 tasks, 30 fps).

Config

• Base VLM: Qwen/Qwen2.5-VL-3B-Instruct (LoRA rank 32, vision encoder frozen)
• Architecture: VLM hidden -> dual AttentiveLatentHead (A + B) -> concat -> ResNetActionHead, action L1 loss (no V-JEPA, no augmentation)
• Action chunk: 30 (@30fps = 1.0s horizon), action dim 7 (joint_0..5 + gripper)
• State: state-free (observation.state zeroed, state_proj frozen)
• Language conditioning: task description string tokenized by the Qwen processor
• Per-task single camera (input renamed to observation.images.camera1):
  • Point at the red cup -> camera_0
  • Point at the blue cup -> camera_2
  • Pick up the blue die and place in the basket -> camera_0
  • Pick up the red die and place in the basket -> camera_2
  • Pull a tissue out of the box -> camera_1
  • Close the laptop -> camera_1
  • Pick up the shoe and stand it upright -> camera_1
• Training: 100k steps, batch 16, lr 5e-5, cosine schedule

Contents

• checkpoints/0XXXXXX/checkpoint.pt — saved every 10k steps. Each is a torch dict: policy, latent_head, free_latent_head, action_head, optimizer, agg_stats, args.
• ur5_qwen25vl_inference.py — self-contained inference (UR5QwenPolicy).
• ur5_policy_server.py — HTTP policy server (/health, /predict, /reset).
• train_ur5_b_qwen25vl.py — training script.

These checkpoints are not standard LeRobot checkpoints; load them with the included UR5QwenPolicy, which rebuilds the VLM + external heads. Requires the lerobot fork with policies/qwen25vl_act on the path (PYTHONPATH=lerobot/src).

Inference (Python)

import numpy as np
from ur5_qwen25vl_inference import UR5QwenPolicy

pol = UR5QwenPolicy("checkpoints/070000/checkpoint.pt", device="cuda")
rgb = np.zeros((480, 640, 3), np.uint8)          # HxWx3 uint8, RGB (task-appropriate camera)
action = pol.select_action(rgb, task="Point at the red cup")   # (7,) joint targets + gripper
# select_action serves one action per call from an internal 30-step queue;
# call pol.reset() at the start of each episode.

Serving (HTTP API)

CUDA_VISIBLE_DEVICES=0 PYTHONPATH=lerobot/src python ur5_policy_server.py \
    --checkpoint checkpoints/070000/checkpoint.pt --host 0.0.0.0 --port 8000 \
    --task "Point at the red cup"

• GET /health -> policy metadata (cameras, action_dim, per-task camera map).
• POST /predict {"images": {"camera_1": "<base64 JPEG>"}, "task": "..."} -> {"action": [7], "latency_ms": ...}.
• POST /reset -> clears the action queue (call at episode start).

Follows ManiUniCon's POLICY_SERVER_API.md. State-free: state is accepted but ignored. The model is language-conditioned and per-task single-camera — feed the camera the task was trained on (see /health.task_camera).