strands-arm-frame_dolly-2026-09-06

strands-arm frame_dolly: the whole leader arm keeps a subject framed and drives the omni base as its dolly

Six servo deltas per 0.1 s (ids 1..6, clipped +-5 deg) plus three base velocity commands (vx, vy, wz in [-5, 5], mapped onto the drive guard caps 0.15 m/s, 0.15 m/s, 0.6 rad/s) from what the Nicla reports (FOMO box centre, size, visible, ToF), the six servo readings, the last command and a shot spec (where in the frame, how big), plus the commanded base velocity (vx, vy, wz). Trained with PPO in a MuJoCo twin of this arm (SO-101 leader + printed head, lens as a camera at the firmware FOV, identified STS3215 servo model, calibrated limits) with occlusion rays, one-frame detector delay, pixel noise and randomized FOV, and the base following the commanded velocity through a ramp; the size spec is a dolly offset the base can deliver (the arm alone cannot: 0.002 size per servo degree, see rl/sim/sensitivity.py). Baseline: the head-only P step watch uses today, arm frozen at the fold.

Top: head-only P. Bottom: this policy, same subject. Left = the Nicla's view (green = shot spec, yellow = detection), right = third person. MP4.

Eval (rl/eval_frame.py, 4 seeds x 64 envs x 15 s, 256 episodes)

PASS: keeps the subject visible more often than the head-only P step. framed = centre error < 0.15 and size error < 0.1. The arm-frozen row is the same network with servos 1..4 held: the difference is what the arm buys. The frame_mobile row is the shipped arm-only cameraman dropped into the same env with the base parked: the difference is what driving buys. Ship rule for this policy: size err < 0.12 and visible > frame_mobile.

Contract

Input obs [1, 33] float32, output act [1, 9] float32 = servo deg per 0.1 s step, ids 1..6; base_*_cmd in [-5, 5] -> [-cap, cap] (0.15 m/s, 0.15 m/s, 0.6 rad/s).

Runner: strands_arm.policy.FramePolicy (image ey flipped to the sim's up-positive frame, size = box height / half frame, ToF mm -> m, targets clamped through kine.clamp_pose). Never run on the real arm yet; shadow mode (real pose, detections and ToF, nothing written) is logged in the repo's RL.md. Source: github.com/cagataycali/strands-arm 56caa5b, rl/envs/frame_env.py, rl/sim/. PPO: {"n_envs": 256, "steps_per_env": 50, "epochs": 4, "minibatches": 8, "lr": 0.0003, "gamma": 0.98, "lam": 0.95, "clip": 0.2, "ent_coef": 0.002, "vf_coef": 0.5, "max_grad_norm": 0.5, "hidden": 128, "act_scale": 5.0, "act_dim": 9, "total_updates": 2500, "device": "cpu", "seed": 0} Env overrides: {"w_act": 0.1, "w_jerk": 0.1, "w_size": 2.0}