Downloads · 30 days
0
edge-inference/hab3-social-rearrange-fabric
hab3-social-rearrange-fabric is a reinforcement learning model from edge-inference. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Trained weights for the Pereason + Go + Fabric multi-agent coordination model on the Social Rearrangement task from Habitat 3.0. Two embodied agents — a Boston Dynamics Spot robot and a humanoid — cooperate to rearran…
Downloads · 30 days
0
Access
Public
Updated May 4, 2026
Repo size
3.6 GB
Likes
0
Public
Click a slice to open those files.
.pth2.2 GB · 100%
From the Hugging Face model README
Trained weights for the Pereason + Go + Fabric multi-agent coordination model on the Social Rearrangement task from Habitat 3.0. Two embodied agents — a Boston Dynamics Spot robot and a humanoid — cooperate to rearrange objects across 37 HSSD scenes.
This model uses Fabric, a learned cross-attention communication module that lets agents exchange compressed multi-modal feature messages each timestep. Fabric fuses semantic (VLM) and geometric (depth) tokens before encoding them into inter-agent messages, enabling richer coordination. Compared to the no-communication Go baseline (69.6% task success), Fabric reaches 76.8% task success with significantly lower collision rates.
This work is part of the thesis "Scalable Multi-Agent Coordination Using a Shared-Context Architecture for Embodied Robotics" by Benjamin Kubwimana. Fabric is the shared-context communication layer explored in the thesis, evaluated here on the Habitat 3.0 social rearrangement benchmark.
| File | Description |
|---|---|
| model-latest.pth | Peak unified Fabric checkpoint (ckpt.250, ~64M frames, 76.8% success) |
| model-semantic-fabric.pth | Semantic-only Fabric baseline (ckpt.69, ~57M frames, 63.9% success) |
Each checkpoint contains the full model state dict for both agents (keys 0, 1) plus training config.
Each episode drops two agents into an HSSD home scene with objects that need to be moved to goal locations. The task is structured as a PDDL planning problem with four subgoal stages:
Full success (pddl_success) requires both agents to complete all subgoals within 750 timesteps.

Both agents use a hierarchical RL policy with three main components:
MessageEncoder (Linear → GELU → Linear → LayerNorm) compresses the fused multi-modal features into 128-dim messagesMessageDecoder (cross-attention with gated residual) decodes the partner's message back into the agent's fused feature space; the semantic portion is then extracted for downstream use| Module | Total params | Trainable params |
|---|---|---|
| SmolVLM2 (VLM) | 303M | 0 (frozen) |
| Depth Anything V2 | 24.8M | 22.1M |
| Go (transformer) | 73.5M | 73.5M |
| Fabric + geo_proj | 6.7M | 6.7M |
| Total | 408.0M | 102.3M |
Oracle navigation + learned manipulation skills (pick, place, nav_to_obj, etc.) using privileged simulator information.
Observations per agent:
is_holding flag| Parameter | Value |
|---|---|
| Total frames | ~64M |
| Batch size | 128 steps x 64 envs |
| Learning rate | 1e-4 (Go), 1e-5 (Fabric, geo_proj, Depth encoder) |
| PPO epochs | 2 |
| Mini-batches | 2 |
| Clip param | 0.2 |
| Discount (gamma) | 0.99 |
| GAE (tau) | 0.95 |
| Entropy coef | 0.001 |
| Max grad norm | 0.5 |
| Trainer | DD-PPO |
| Fabric msg_dim | 128 |
| Fabric num_heads | 4 |
| geo_proj pooled tokens | 128 |
| Metric | Value |
|---|---|
| Task success | 76.8% |
| Reward | 25.5 |
| Stage 1.1 (Spot picks) | 95.8% |
| Stage 1.2 (Spot places) | 87.2% |
| Stage 2.1 (Human picks) | 95.2% |
| Stage 2.2 (Human places) | 83.0% |
| Collision rate | 10.7% |
| Cooperation reward | +2.79 |
| Metric | Go — No Comm | Fabric | Delta |
|---|---|---|---|
| Task success | 69.6% | 76.8% | +7.2pp |
| Reward | 23.6 | 25.5 | +1.9 |
| Collision rate | 15.4% | 10.7% | -4.7pp |
| Cooperation reward | +2.77 | +2.79 | +0.02 |
| Frames | Reward | Task success | Collisions | Cooperation |
|---|---|---|---|---|
| 0M | 1.0 | 0.0% | 100.0% | -0.50 |
| 10M | 9.2 | 5.5% | 25.7% | +0.11 |
| 25M | 12.3 | 13.6% | 19.3% | +0.44 |
| 45M | 24.1 | 72.2% | 12.5% | +2.66 |
| 50M | 24.5 | 73.0% | 11.6% | +2.72 |
| 64M | 25.5 | 76.8% | 10.7% | +2.79 |
Requires Habitat 3.0 (v0.3.3) with habitat-baselines, habitat-sim, and the Pereason+Go+Fabric policy classes.
python -u -m habitat_baselines.run \
--config-name=social_rearrange/pereason_go_fabric \
habitat_baselines.evaluate=True \
habitat_baselines.eval.should_load_ckpt=True \
habitat_baselines.eval_ckpt_path_dir=model-latest.pth \
habitat_baselines.test_episode_count=50 \
habitat_baselines.num_environments=1 \
habitat.dataset.data_path=data/datasets/hab3_episodes/val/social_rearrange.json.gz \
habitat.dataset.scenes_dir=data/scene_datasets \
'habitat_baselines.eval.video_option=["disk"]'
hab3_episodes dataset@mastersthesis{kubwimana2026scalable,
title = {Scalable Multi-Agent Coordination Using a Shared-Context Architecture for Embodied Robotics},
author = {Kubwimana, Benjamin},
year = {2026},
school = {Georgia Institute of Technology},
note = {Model weights: \url{https://huggingface.co/edge-inference/hab3-social-rearrange-fabric}}
}
Built on the Habitat 3.0 platform:
@inproceedings{puig2023habitat3,
title = {Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots},
author = {Puig, Xavier and Undersander, Eric and Szot, Andrew and Cote, Mikael Dallaire and Batra, Dhruv and Berges, Vincent-Pierre and others},
booktitle = {ICLR},
year = {2024}
}
MIT. The underlying Habitat platform and HSSD scenes have their own licenses — see the Habitat 3.0 repo for details.