parol-smolvla-simulation-weights

PAROL6 SmolVLA — language-conditioned pick-and-place (simulation)

A SmolVLA finetune that drives a simulated PAROL6 6-DoF arm in MuJoCo. Two cubes sit on a table, red and blue, next to a container. A plain-text instruction selects which cube to move.

This checkpoint solves the task 32% of the time from pixels, but it does not reliably read the instruction. Read Results before using it — the honest summary is that the manipulation works and the language grounding mostly does not.

What this is

The instruction is one of ten phrasings, five per colour, e.g. "Put the red cube in the box.", "Drop the blue block in the bin."

Results

Every number below is on held-out seeds that were never used for checkpoint selection, except where marked.

The instruction-swap ablation

The measurement that matters. Each scene is run twice — identical cube positions, identical pixels — told "red" once and "blue" once — recording which cube actually moved.

Interpret this as weak-to-absent language conditioning. A policy that ignores the sentence and picks a cube at random would switch on roughly half of scenes by luck and be correct both ways about a quarter of the time. The measured 33% / 30% is near that chance level.

So the policy has learned the visuomotor task — approach, grasp, transport, release, at a median 286 steps against the oracle's 218 — while the language channel contributes little. It is a competent cube-placer that is not listening.

Why the previous attempt scored zero

An earlier 20,000-step × batch 4 run (0.64 epochs — under one pass over the data) scored 0/20 held-out and 0/12 on the swap. That 0/12 was not evidence about language: the policy could barely move a cube at all, so the ablation conflated "does not read the instruction" with "cannot do the task". Only once success was non-zero could conditioning be measured.

Scaling to 10.25 epochs took success from 0% to 32% and then plateaued — validation success flattened around step 14,000 and training loss around step 26,000. More training on this recipe will not help. The remaining gap is architectural: the finetune runs with freeze_vision_encoder=true and train_expert_only=true, so only the action expert is trained and nothing adapts the language pathway to this task.

Usage

Requires LeRobot 0.6.1 and the parol6-sim environment package.

from lerobot.configs.policies import PreTrainedConfig
from lerobot.policies.factory import get_policy_class, make_pre_post_processors

cfg = PreTrainedConfig.from_pretrained("akoniti/parol-smolvla-simulation-weights")
policy = get_policy_class(cfg.type).from_pretrained(
    "akoniti/parol-smolvla-simulation-weights")

The normalisation statistics live in the saved preprocessor, not in the weights. A policy invoked without policy_preprocessor / policy_postprocessor runs happily and emits garbage. Load them.

The task string is required and must be non-empty — this is a VLA, and an empty instruction is silently accepted while destroying the result.

Files

Limitations

• Simulation only. Never validated on physical hardware, and not intended to be run on a real arm without further work.
• Weak language grounding, as measured above.
• Joint-velocity limits are exceeded on some episodes. Peak velocity reached 2.07× the published limit for J5 on the swap evaluation, over 214 of 188,560 physics steps. MuJoCo models no step loss so this does not affect simulated success; on a real stepper-driven arm it would matter.
• Cube sampling is confined to one region of the workspace; behaviour outside it is unmeasured.
• Evaluation is mildly non-deterministic across machines — the same checkpoint has scored one episode apart on different Macs, from MPS floating-point differences.

Reproducing

The dataset is not distributed; it regenerates exactly from seeds, and has been verified to reproduce frame-for-frame across three different machines. The full runbook — clone, bootstrap, regenerate, sweep throughput, train, evaluate — is SETUP.md in the project repository.

Training took ~34.5 h on an Apple M5 Max (36 GB unified) at batch 32. Batch size must be re-measured per machine: on this hardware throughput collapsed 12× at batch 64 and ran out of memory at 128.