impact_so101_tape

IMPACT — SO-101 tape task

Instruction-Modulated Perception + ACTion chunking: ACT with a language tower. Two camera frames, a 6-dim joint state and a natural-language instruction go in; a 50-step action chunk comes out in one forward pass.

The instruction enters twice — as 32 tokens appended to the DETR encoder sequence, and as FiLM scale/shift on the ResNet-18 stages, so language modulates perception rather than only the fused memory.

⚠️ This checkpoint cannot demonstrate instruction following

It is trained on a single task — khanhnd61/so101-tape, one instruction ("Put the tape into the box") on every frame. With no instruction variation there is no gradient signal asking the language pathway to do anything, and the FiLM head barely moves off its zero initialization (‖W‖∞ = 0.0015 after 4000 steps).

So: this checkpoint exercises the architecture end to end and gives the CPU port a real checkpoint to be validated against. It is not evidence that IMPACT uses language. That claim needs a multi-task dataset whose tasks share a scene and differ only by instruction, plus the use_film=false and language_dropout=1.0 ablation arms. Do not read the loss curve as instruction following.

Architecture

chunk_size is halved because the point of a policy you can talk to is retasking it, and at 30 Hz a 100-step chunk means a new instruction takes up to 3.33 s to take effect.

The frozen T5-small tower is not in this checkpoint — it is fully determined by google-t5/t5-small and re-fetched on load, so model.safetensors holds the 78M trainable tensors. load_state_dict reports those keys as missing; that is expected.

Training

lerobot-train --policy.type=impact \
  --dataset.repo_id=khanhnd61/so101-tape_20260804_224429 \
  --batch_size=8 --steps=4000 --save_freq=1000 --policy.device=cuda

3434 frames / 10 episodes, ~2.7 step/s on an RTX 3060 (≈25 min). Final l1_loss 0.146.

CPU inference

Runs on CPU through vla.simd's tcpu_impact engine. Query latency (median warm, fp32):

Stall = latency ÷ executed motion. Below 1.0 the policy keeps up with the chunk it emits; IMPACT is the only language-conditioned policy in that engine that closes the loop on a Pi 5. Latency is weight-independent, so these hold for this checkpoint.

The C++ engine is validated against this exact checkpoint: tokenization bit-exact, FiLM γ/β to 1.3e-07, action chunk to 1.5e-06 max abs against an action magnitude of 1.364 — the fp32 noise floor.