ur5_qwen25vl_act_chunk30_state

Ngseo/ur5_qwen25vl_act_chunk30_state

Qwen2.5-VL-Act (Option-B) fine-tuned on Ngseo/ur5_teleop_multitask (UR5 teleop, 350 eps, 7 tasks, 30 fps). This is the checkpoint that was served on the policy endpoint (:8000).

Config

• Base VLM: Qwen/Qwen2.5-VL-3B-Instruct (LoRA r32 on LM q_proj/v_proj and vision qkv; vision base frozen)
• Architecture: VLM hidden -> dual AttentiveLatentHead -> concat -> ResNetActionHead, action L1 loss
• Action chunk: 30 (@30fps = 1.0 s), action dim 7 (joint_0..5 + gripper)
• State: USED — observation.state (7 joint values) fed to the model, MEAN_STD normalized (state_proj trainable)
• Domain randomization: none
• Language: task description string (Qwen processor)
• Per-task single camera (renamed to observation.images.camera1): red-cup->camera_0, blue-cup->camera_2, blue-die->camera_0, red-die->camera_2, tissue/laptop/shoe->camera_1
• Training: 100k steps, batch 16, lr 5e-5, cosine

Contents

• checkpoints/0XXXXXX/checkpoint.pt — every 10k steps (10k..100k). torch dict: policy, latent_head, free_latent_head, action_head, optimizer, agg_stats, args. (090000 was the served endpoint checkpoint.)
• ur5_qwen25vl_inference.py — UR5QwenPolicy (rebuilds VLM + heads, handles LoRA-vision & state, un-normalizes action).
• ur5_policy_server.py — HTTP server (/health, /predict, /reset).
• train_ur5_b_qwen25vl.py — training script.

Not a standard LeRobot checkpoint — load with the included UR5QwenPolicy (needs the lerobot fork with policies/qwen25vl_act, PYTHONPATH=lerobot/src).

Inference

import numpy as np
from ur5_qwen25vl_inference import UR5QwenPolicy
pol = UR5QwenPolicy("checkpoints/100000/checkpoint.pt", device="cuda")
rgb = np.zeros((480,640,3), np.uint8)                 # task-appropriate camera
state = [j0,j1,j2,j3,j4,j5,gripper]                   # raw radians; normalized internally
action = pol.select_action(rgb, task="Point at the red cup", state=state)  # (7,)

State is required for this model (state-conditioned). Serve with ur5_policy_server.py; clients POST {"state":[...7...], "images":{"camera_1":"<b64 JPEG>"}, "task":"..."} to /predict.