lehome-groot-submission-3

LeHome Challenge 2026 — Submission

This HuggingFace repository contains our submission to the LeHome Challenge 2026 (Garment Manipulation Skill Learning in Household Scenarios). It is a derivative of the official lehome-challenge starter code, fine-tuned with GR00T N1.5 via the LeRobot training framework.

It is intended to be self-contained — you can clone or download this single HF repo and reproduce our evaluation end-to-end without any external code source. The same source tree is also attached as a zip in our Google form submission; both contain identical contents.

The original challenge README is preserved at lehome_challenge_README.md.

---

1. Submission Summary

Detailed per-episode and per-garment results: submission/rollout_results.txt.

---

2. Prerequisites

NVIDIA Omniverse / IsaacSim requires accepting an EULA. Our shs/evals/eval_groot.sh exports the required environment variables (OMNI_KIT_ACCEPT_EULA=YES, ACCEPT_EULA=Y, PRIVACY_CONSENT=Y) before launching the simulator, so no manual prompt should appear.

---

3. Quick Start (Evaluator's Path)

# 1) Get the submission tree.
#    Option A — pull straight from HuggingFace (source code + checkpoint + metadata):
huggingface-cli download theo-zhou/lehome-groot-submission-3 \
    --exclude "pretrained_model/*" --exclude "dataset_meta/*" \
    --local-dir lehome-challenge
cd lehome-challenge

#    Option B — extract the zip we attached in the Google form submission, then `cd` in.

# 2) Set up the environment with uv
uv sync
source .venv/bin/activate

# 3) Install extra build/runtime packages used by the GR00T policy stack
uv pip install packaging ninja
uv pip install flash-attn --no-build-isolation

# 4) Clone and install IsaacLab
mkdir -p third_party
git clone https://github.com/lehome-official/IsaacLab.git third_party/IsaacLab
./third_party/IsaacLab/isaaclab.sh -i none

# 5) Install the local LeHome package in editable mode
uv pip install -e ./source/lehome

# 6) Download challenge assets (garments, scenes, robots)
huggingface-cli download lehome/asset_challenge --repo-type dataset --local-dir Assets

# 7) Download our checkpoint into the path the eval script expects
huggingface-cli download theo-zhou/lehome-groot-submission-3 \
    --include "pretrained_model/*" \
    --local-dir outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/009000/

# 8) Download per-category dataset metadata and stage it.
#    HF download preserves repo-relative paths inside --local-dir, so files
#    land at Datasets/_dataset_meta_staging/dataset_meta/<cat>_merged/meta/.
#    We then mv each meta/ into the LeRobot dataset_root layout
#    (<dataset_root>/meta/) and clean up the staging dir.
huggingface-cli download theo-zhou/lehome-groot-submission-3 \
    --include "dataset_meta/*" \
    --local-dir Datasets/_dataset_meta_staging

for cat in top_long top_short pant_long pant_short; do
    mkdir -p Datasets/example/${cat}_merged
    mv Datasets/_dataset_meta_staging/dataset_meta/${cat}_merged/meta \
       Datasets/example/${cat}_merged/meta
done
rm -rf Datasets/_dataset_meta_staging

# 9) (Recommended) Run a smoke test before the full eval (~5-15 min on CPU)
xvfb-run -a python -m scripts.eval \
    --policy_type lerobot \
    --policy_path outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/009000/pretrained_model \
    --garment_type top_long \
    --dataset_root Datasets/example/top_long_merged \
    --num_episodes 1 \
    --max_steps 200 \
    --enable_cameras --headless --device cpu \
    --log_suffix smoke

# 10) Snapshot our self-reported results before re-running the full eval
cp submission/rollout_results.txt /tmp/ours.txt

# 11) Run the full Release evaluation (4 categories × 12 garments × 2 episodes)
bash shs/evals/eval_groot.sh

# 12) Aggregate YOUR fresh logs into submission/rollout_results.txt and diff
bash shs/submit/roll_out_txt_generation.sh          # MODE=self (default)
diff /tmp/ours.txt submission/rollout_results.txt   # see "Reproducibility" note below

# 13) (Optional) Sanity-check that OUR shipped reference logs reproduce
#     our self-reported numbers via the same aggregator:
MODE=ours bash shs/submit/roll_out_txt_generation.sh
diff /tmp/ours.txt submission/rollout_results.txt   # diff is metadata-only:
                                                    #   - regeneration timestamp ("Generated:")
                                                    #   - per-category "Log mtime:" reflects when you ran step 13
                                                    # all numeric metrics should match exactly

Note on huggingface-cli vs hf: We use the legacy huggingface-cli name throughout the README because it is what ships with the huggingface_hub version pinned in uv.lock. If you have upgraded to huggingface_hub>=1.0, the equivalent commands are hf download ....

---

4. Detailed Setup

4.1 Environment

We use uv to manage the Python 3.11 environment. IsaacLab is installed from the official LeHome fork under third_party/, and the local source/lehome package is installed in editable mode.

uv sync
source .venv/bin/activate

# Extra packages used during our environment setup.
uv pip install packaging ninja
uv pip install flash-attn --no-build-isolation

# IsaacLab from the LeHome fork.
mkdir -p third_party
git clone https://github.com/lehome-official/IsaacLab.git third_party/IsaacLab
./third_party/IsaacLab/isaaclab.sh -i none

# Local LeHome package.
uv pip install -e ./source/lehome

uv sync installs the exact versions listed in uv.lock, including isaacsim==5.1.0 and lerobot==0.4.3. The additional commands above match the environment used during training and the evaluations whose results are reported here.

4.2 Simulation Assets

The challenge garments, scenes, and robots come from the official lehome/asset_challenge HuggingFace dataset. They are not redistributed in our submission.

huggingface-cli download lehome/asset_challenge --repo-type dataset --local-dir Assets

4.3 HuggingFace Repo Layout

The full submission — source code, trained checkpoint, dataset metadata, and reference eval logs — is hosted at theo-zhou/lehome-groot-submission-3. Top-level layout on the Hub:

theo-zhou/lehome-groot-submission-3
├── README.md                             # this file (also rendered as the model card)
├── LICENSE
├── pyproject.toml / uv.lock              # pinned environment definition
├── .python-version
├── lehome_challenge_README.md            # original challenge README, preserved
├── configs/                              # training config (train_groot.yaml)
├── source/lehome/                        # custom IsaacLab simulation env
├── scripts/                              # eval entrypoint + policy adapters + aggregator
├── shs/                                  # eval / submit shell drivers
├── submission/
│   ├── rollout_results.txt               # our self-reported per-episode numbers
│   └── reference_logs/<category>/*.log   # exact eval logs that produced rollout_results.txt
├── pretrained_model/                     # ~6.96 GB GR00T-N1.5 fine-tuned weights
│   ├── config.json
│   ├── model.safetensors
│   ├── policy_postprocessor.json
│   ├── policy_postprocessor_step_0_groot_action_unpack_unnormalize_v1.safetensors
│   ├── policy_preprocessor.json
│   ├── policy_preprocessor_step_2_groot_pack_inputs_v3.safetensors
│   └── train_config.json
└── dataset_meta/                         # LeRobot dataset metadata (no parquet/video)
    ├── top_long_merged/meta/
    ├── top_short_merged/meta/
    ├── pant_long_merged/meta/
    └── pant_short_merged/meta/

pretrained_model/ and dataset_meta/ are kept separate from the source tree so they can be downloaded selectively via --include / --exclude flags (see steps 1, 4, 5 in §3 Quick Start).

Download:

# Checkpoint into the location the eval script expects
huggingface-cli download theo-zhou/lehome-groot-submission-3 \
    --include "pretrained_model/*" \
    --local-dir outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/009000/

# Per-category dataset metadata. Staged under Datasets/_dataset_meta_staging/
# (HF download preserves repo paths inside --local-dir), then moved into the
# LeRobot dataset_root layout (<dataset_root>/meta/).
huggingface-cli download theo-zhou/lehome-groot-submission-3 \
    --include "dataset_meta/*" \
    --local-dir Datasets/_dataset_meta_staging

for cat in top_long top_short pant_long pant_short; do
    mkdir -p Datasets/example/${cat}_merged
    mv Datasets/_dataset_meta_staging/dataset_meta/${cat}_merged/meta \
       Datasets/example/${cat}_merged/meta
done
rm -rf Datasets/_dataset_meta_staging

The dataset metadata is used by LeRobotDatasetMetadata to load feature schemas; the policy's normalization parameters are baked into its own preprocessor safetensors and do not depend on the dataset stats.

---

5. Running Evaluation

The end-to-end evaluation entrypoint is shs/evals/eval_groot.sh. It iterates over the four garment categories (top_long, top_short, pant_long, pant_short), and for each category runs scripts.eval against the full Release garment list in Assets/objects/Challenge_Garment/Release/.

bash shs/evals/eval_groot.sh

Defaults baked into the script:

Logs are written to logs/<garment_type>/<MM-DD_HH-MM-SS>_eval_<log_suffix>.log. Each category produces one log file containing per-episode Return / Length / Success, a per-garment summary, and an overall summary block. The reference logs shipped with this submission were generated with 2 episodes per garment, for 96 total episodes across the four categories.

Wall-clock time: With both simulation and policy on CPU, a full run with the default settings (96 episodes = 4 categories × 12 garments × 2 episodes) takes roughly 8-12 hours total on a typical workstation. If a CUDA-capable GPU is visible and the installed PyTorch is a CUDA build, the policy will automatically run on GPU — see scripts/utils/evaluation.py:242 (device = "cuda" if torch.cuda.is_available() else "cpu"); no CLI flag is required. The --device cpu in shs/evals/eval_groot.sh is IsaacLab's AppLauncher flag and only controls the simulator, which the challenge framework requires to be CPU.

5.1 Reproducibility Note

Each episode samples a random initial garment pose, so per-episode Return / Length / Success values will differ between runs. We do not pin a seed because the official challenge protocol uses random initialization.

When comparing against submission/rollout_results.txt, focus on the aggregate metrics:

• per-garment success_rate
• per-category success_rate (overall)
• the 4-category Overall Summary success rate

These should be statistically close to ours, but note that the shipped reference logs and the default evaluation script use 2 episodes per garment. Increase NUM_EPISODES in shs/evals/eval_groot.sh if you want tighter agreement. Wall-clock time scales linearly with the value.

---

6. Generating the Rollout Report

shs/submit/roll_out_txt_generation.sh is a thin wrapper around scripts/aggregate_rollout_results.py and supports two modes via the MODE env var.

MODE=self (default) — your own freshly-generated logs

bash shs/submit/roll_out_txt_generation.sh

Picks the latest log per category under logs/<category>/ matching --log_suffix groot-n1.5-batch64-9k (the suffix written by shs/evals/eval_groot.sh) and writes the aggregated report to submission/rollout_results.txt.

MODE=ours — the reference logs we shipped

MODE=ours bash shs/submit/roll_out_txt_generation.sh

Aggregates the four log files we shipped under submission/reference_logs/<category>/ — these are the exact logs that produced the numbers in our self-reported submission/rollout_results.txt. Useful for sanity-checking that the aggregator deterministically reproduces our reported numbers from the raw logs (the diff against a snapshot of the shipped rollout_results.txt should be empty modulo the regeneration timestamp at the top).

Underlying behavior

In both modes the aggregator:

1. Picks the latest log per category matching --log_suffix groot-n1.5-batch64-9k under the resolved --logs_dir. Explicit per-category --<category>_log overrides are also supported by the Python script directly.
2. Parses per-episode results, per-garment summaries, and overall metrics directly from the logs.
3. Auto-extracts metadata (policy_type, policy_path, action_dim, episodes_per_garment, eval list source) from each log header.
4. Overwrites submission/rollout_results.txt in place — snapshot it first (cp submission/rollout_results.txt /tmp/ours.txt) if you want to diff against our self-reported numbers afterward.

---

7. Repository Layout

---

8. Submission Identifier

• HuggingFace repo: https://huggingface.co/theo-zhou/lehome-groot-submission-3
• Submission tag: submission_3
• Source code is also attached as a zip (submission_3.zip) in our Google form submission; both are byte-for-byte identical to this repo's source tree (excluding the pretrained_model/ and dataset_meta/ folders which are only on HuggingFace due to size).

---

9. Acknowledgments

This submission builds directly on:

• Isaac Sim and Isaac Lab for the simulation backend.
• LeRobot for the training and policy loading infrastructure.
• NVIDIA GR00T N1.5 for the pretrained vision-language-action backbone.
• The LeHome Challenge organizers for the starter code and benchmark.