lehome-bi-so101-gr00t

LeHome Challenge — bi-SO101 GR00T 1.7 policy

This Hugging Face repository is the submission package for the LeHome 2026 garment-folding challenge. It ships a prebuilt Docker image containing a finetuned GR00T 1.7 policy and the HTTP bridge that implements the upstream DockerPolicy contract (POST /reset, POST /infer). The container consumes the observation dict produced by lehome-challenge/scripts/eval.py and returns 12-DoF action chunks for the two SO101 arms.

Evaluators do not need to build anything. The flow is:

1. Download the image tarball from this repo
2. docker load it
3. docker run it, then run the organizer's evaluator against http://localhost:8080

The Cosmos backbone, the finetuned checkpoint, and all Python dependencies are already baked into the image, so the runtime container needs no network access and no Hugging Face token.

Prerequisites

• Linux x86_64
• NVIDIA GPU with >=24 GB VRAM
• NVIDIA driver >= 525
• Docker 20.10+
• NVIDIA Container Toolkit (required for --gpus all)
• a checkout of the organizer-provided lehome-challenge repository for running evaluation
• ~30 GB of free disk for the image

1. Download the Docker image

The image is published on this Hugging Face repo as a single tarball (lehome-gr00t.tar, tracked with Git LFS). Pull it with whichever method you prefer:

# Option A — huggingface-cli (recommended)
pip install -U "huggingface_hub[cli]"
huggingface-cli download alex-luci/lehome-bi-so101-gr00t \
    lehome-gr00t.tar \
    --local-dir .

# Option B — git lfs
git lfs install
git clone https://huggingface.co/alex-luci/lehome-bi-so101-gr00t
cd lehome-bi-so101-gr00t
git lfs pull --include="lehome-gr00t.tar"

2. Load the image into Docker

sudo docker load -i lehome-gr00t.tar

This registers the image locally as lehome-gr00t:hf. Verify:

sudo docker images lehome-gr00t:hf

3. Run the policy server

Start the container and expose the bridge on port 8080:

sudo docker run --rm --gpus all --ipc=host \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    -p 8080:8080 \
    lehome-gr00t:hf

Wait for the server log:

[bridge] listening on 0.0.0.0:8080 → in-process /model

Optional smoke test from another shell:

nc -zv localhost 8080

4. (Optional) Run with a Hugging Face token

The image is fully self-contained: the Cosmos backbone, the finetuned checkpoint, and all Python dependencies are baked in, so under normal conditions no Hugging Face token is required at runtime.

If for any reason the container needs to pull something from the Hugging Face Hub (for example, a transient cache miss or a future variant of the image that expects an online fetch), you can pass a token via the HF_TOKEN environment variable.

Get a Hugging Face token

1. Sign in at https://huggingface.co (create a free account if needed).
2. Go to Settings → Access Tokens: https://huggingface.co/settings/tokens
3. Click New token, give it a name (e.g. lehome-eval), and select the Read role.
4. Copy the generated token (hf_...). Treat it like a password.

Accept the Cosmos model license

The GR00T 1.7 backbone depends on NVIDIA's Cosmos model, which is gated on Hugging Face. Before the token can be used to fetch it, you must visit the model page and accept the access terms:

• https://huggingface.co/nvidia/Cosmos-Reason2-2B

On that page you will see a banner stating "You need to agree to share your contact information to access this model". Click through and accept — once your account is approved, your HF_TOKEN will be authorized to download the weights.

Run the container with the token

export HF_TOKEN=hf_xxx_your_token_here

sudo docker run --rm --gpus all --ipc=host \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    -p 8080:8080 \
    -e HF_TOKEN="$HF_TOKEN" \
    lehome-gr00t:hf

This step is optional — only use it if a run without the token fails with a Hugging Face authentication or download error.

Training data — synthetic generation pipeline

The challenge ships a direct Isaac Lab environment for evaluation. Hand-collecting ~200 demonstrations × 40 garments was not feasible, so we built a synthetic data generator on top of MimicGen that turns a small set of human teleop demos into a large, diverse training set. Adapting MimicGen to deformable garments and a dual-arm SO-101 setup required several non-trivial extensions:

Reference implementation: https://github.com/REBELDOT-SOLUTIONS-S-R-L/ROBOTICS-lehome-challenge

In practice, we recorded 640 human teleoperation demonstrations across 10 seed garments (64 demos per seed garment), then used those seed datasets to generate ~2,600 successful synthetic episodes covering 28 of the 40 challenge garments. Similar garments within a category reused the same seed teleop bank, so we did not need to hand-record all 40 garments individually.

• Hybrid env (direct + manager-based). The original garment sim is a direct Isaac Lab environment, but MimicGen only operates on manager-based envs. We kept the original garment loading logic, physics, and environment assets in the direct env, then wrapped them in a manager-based shell that adds the observations, terminations, recorders, and Mimic env configs needed for data generation. That avoided rewriting the garment simulator while still making it compatible with the MimicGen pipeline. The manager shell also exposes cloth-specific subtask observations, including garment-aligned release zones derived from the four corner keypoints, so annotation and generation can use geometric checks instead of brittle replay heuristics.
• Keypoint-anchored matching for deformables. MimicGen assumes rigid reference frames. Garments are point clouds, so we replaced rigid-object matching with 6 semantic garment keypoints in world space: left_upper, left_middle, left_lower, right_upper, right_middle, right_lower. For tops, these roughly correspond to neck / sleeve / waist structure; for pants, waist / knee / foot structure. The default Isaac Lab strategy only matches on one point, which is not enough for cloth because the same target keypoint can require a very different grasp when the garment is rotated. We therefore added a fixed-per-subtask 2–4 keypoint nearest-neighbor selection strategy, so source demos are chosen from garments with similar orientation, not just similar single-point position. For example, left-sleeve grasping on tops matches on garment_left_middle, garment_left_upper, garment_left_lower, while some pants transfers can match on just garment_right_lower, garment_right_upper.
• Dual-arm synchronization. Each arm is allowed to source its trajectory from a different demo via per-arm source selection, which improves reuse and success rate because one arm often needs a different demonstration than the other. The problem is that those demonstrations have different lengths, so without extra constraints the arms drift out of sync and pull the cloth at the wrong times. We added explicit synchronization constraints at critical subtask boundaries: for tops, before the simultaneous waist re-grasp / fold stage; for pants, before the symmetric two-leg fold stage; and before the return-home phase. If one arm arrives early, it holds its pose until the other arm reaches the paired boundary.
• Real-time annotation. Garment dynamics are stochastic — replaying a successful episode to label it offline often fails because the cloth behaves differently on replay. Instead of record-then-replay annotation, we evaluate the subtask conditions in-loop while recording: EEF-to-keypoint distance for grasps, garment-aligned release-zone entry for transfers, middle-keypoint height for releases, lower-to-upper keypoint distances for fold completion, waiting-position reached, and final fold success. This required a custom recorder, because the default recorder does not save generation-ready datagen_info. Our recorder writes per-step observations/actions, subtask signals, both arms' joint states, initial garment and arm states, and per-episode datagen_info including the semantic garment keypoints. In practice this cut operator time roughly in half to one third, because demos no longer had to be replayed for annotation.
• Halton-sequence state coverage. Random init left holes in the (pos_x, pos_y, rot_x, rot_y) state space, so generated rollouts often started outside the demo region. We replaced random recording resets with a deterministic 4D Halton sequence over the category-specific soft-reset ranges for (pos_x, pos_y, rot_x, rot_y). Each seed garment was recorded over 64 Halton poses, always retrying failed attempts instead of skipping them. The main benefit was better spatial coverage of the demonstration bank across the table: instead of having recording sessions that happened to cluster near the center, we forced demonstrations to cover the full usable workspace so MimicGen could find nearby source demos for off-center starts.
• Better IK + tip-defined EEF. Default IK failed on parts of the workspace; the differential IK solver struggled especially when the EEF moved close to the robot base or into targets near joint-angle limits. We integrated Pinocchio as a more reliable IK backend for generation, and also used it when converting recorded joint trajectories to EEF-space trajectories. We also re-anchored the EEF frame from the gripper center to a tip xform, which improved XY placement, Z placement, and orientation alignment during cloth contact.
• Grasp reliability. Removed unnecessary collision capsules around the gripper, added a per-grasp downward trajectory offset (so MimicGen's augmentation can't lift the EEF off the table), and disabled action augmentation on the gripper open/close channel. Concretely, the default robot USD had two collision capsules around each gripper tip; we disabled them at runtime because they interfered with reliable cloth contact. For grasp subtasks, we find the lowest point of the source trajectory, measure its offset from the table, and shift the generated trajectory downward by that amount so the gripper actually reaches the cloth. For the gripper channel, we stopped interpolating toward the next waypoint value and instead held the last source-demo value between waypoints. Together, these fixes eliminated three common failure modes: missed grasps, grasps that happened too high above the cloth, and early release / incomplete closure caused by gripper interpolation.

The final training set used only successful synthetic rollouts and covers all four garment categories (top_long, top_short, pant_long, pant_short). The GR00T 1.7 checkpoint baked into this image was finetuned on that dataset.

Dataset tooling — rebelHDF5

The training pipeline runs entirely on HDF5 files (Isaac Lab + MimicGen output). Open-source MyHDF5 was a good starting point for raw inspection but didn't fit robotics workflows, so we forked it into rebelHDF5 and added the features we actually needed to iterate on dataset quality.

Project link: https://github.com/alex-luci/myhdf5/tree/pose-trace-integration

Main view: opened-files panel on the far left, HDF5 group/demo tree, and a tabular view of the selected dataset (here, an actions array). The same chrome wraps every other tool below.

• Pose Trace. A 3D viewer that renders the EEF trajectory as a continuous path alongside the garment keypoints over time, so failed grasps, drift, or misaligned trajectories are spotted at a glance instead of from raw arrays. A second tab plots the supporting metrics — EEF-to-keypoint distances, garment fold-form distances, and EEF-vs-keypoint heights over the episode.

  

  3D Pose Trace: left-arm (blue) and right-arm (red) EEF paths plotted with the six garment keypoints over time. The bottom slider scrubs through the episode step-by-step.

  

  Pose Trace analytics tab: per-step EEF-to-keypoint distances (top), garment fold-form distances (middle), and EEF-vs-keypoint heights (bottom), broken out for the left and right arms.

• Video Converter. Extracts the camera streams from an HDF5 episode and produces an MP4 directly in the browser — no external scripts, click-to-preview qualitative review of any sample.

  

  Video Converter: top-down camera stream extracted from the selected episode and played back inline at native FPS, with a Save Video button to export an MP4.

• Data Processing. In-browser cut / merge / append operations on episodes, with key-level selection so you can keep only the parts of an episode you want. Bad rollouts get pruned, multiple generation runs are merged, new batches are appended — all without leaving the viewer.

  

  Data Processing: Cut operation across an episode range (demo_0–demo_99) with key-level selection — pick exactly which dataset paths (e.g. initial_state/articulation/left_arm/joint_position) survive into the output file.

• Cloth Distribution Analysis. Scatter of init poses (success vs failed overlaid on the teleop set) plus failure-rate heatmaps over the (pos_x, pos_y) and (rot_x, rot_y) init space. This is what closes the loop with the Halton sampler — failure-prone regions become directly visible and can be re-sampled. Clicking any generated demo on the scatter draws an arrow back to the source teleop demo MimicGen pulled from, so failure clusters can be traced to specific source demonstrations.

  

  Init-pose scatter: every generated demo plotted on the (init_pose_x, init_pose_y) plane, color-coded as success (green), failed (red), or teleop source (blue), so coverage and failure clustering are visible at a glance.

  

  Click-to-trace: selecting a generated demo on the scatter draws arrows to the source teleop demo(s) MimicGen pulled from, with a tooltip listing the source path — so a failure cluster can be tied back to specific source demonstrations.

  

  Failure-rate heatmap over the init-position space (pos_x, pos_y), aggregated from generated demos via nearest-neighbor pooling. Hotter cells = higher failure rate — directly drives where to oversample next.

  

  Same heatmap projection over the init-rotation space (rot_x, rot_y), exposing orientation regimes the policy struggles with independently of position.

• Databricks integration. Single interface for uploading/downloading datasets, managing secrets, and triggering pipelines, so local iteration and cloud runs share one workflow.

  

  Screenshot 1 (Databricks page, top): the secrets and job-control section, where API tokens, the Databricks workspace URL, and rollout-step settings are pushed as scoped secrets; the same page also uploads the opened HDF5 file and triggers named jobs such as synthetic-data generation, HDF5→LeRobot conversion, and training.

  

  Screenshot 2 (Databricks page, bottom): the Volume Browser, which lets you inspect arbitrary Databricks volume paths such as trained_models/ and pull selected files back to a configurable local download directory with one click.

Net effect: HDF5s stopped being opaque archives and became interactive dataset assets, which is what made the synthetic pipeline above tractable to debug at scale.

What's in this repo

.
├── lehome-gr00t.tar          # prebuilt Docker image (Git LFS) — what evaluators run
├── checkpoint/               # finetuned GR00T 1.7 model weights + configs
├── docker/                   # container entrypoint (baked into the image)
├── scripts/                  # HTTP bridge used by the evaluator (baked into the image)
├── screenshots/              # figures referenced in this README
├── README.md                 # submission instructions + technical overview
├── rebelhdf5.md              # extended write-up for dataset tooling
└── synthetic-data-generator.md
                              # extended write-up for the data pipeline

Troubleshooting

docker load fails with "unexpected EOF" or a checksum error — the tarball did not download fully. Re-pull it via huggingface-cli or git lfs pull (Git LFS objects are not fetched by a plain git clone).

could not select device driver "" with capabilities: [[gpu]] — install the NVIDIA Container Toolkit and restart Docker.

Out-of-memory on the GPU at startup — GR00T 1.7 inference needs real VRAM (>=24 GB recommended). Stop other GPU users first.

Server up but eval times out — make sure no firewall is between the eval process and localhost:8080. nc -zv localhost 8080 should succeed.