fastwam-trex62-prefilling-icl

FastWAM T-Rex 62-D — Prefilling ICL

This is the prefilling FastWAM-ICL checkpoint trained on the pick_fruits subset of wushr-lance/VLA2Vec. Both ICL variants start from the exact same FastWAM T-Rex baseline at step 27,708 and use the same manually audited, fixed EgoDex reference.

Release status: validated training completion; convergence gate passed. The checkpoint reached the planned 5,480 optimization steps. This is a research checkpoint and has not yet been assigned a physical-robot success rate.

Checkpoints

Final ICL optimization step: 5,480. The published checkpoint is a full consolidated inference-weight file.

ICL topology

The 32x1024 Context Tokens are prefilled into the FastWAM context, and the model is fine-tuned end to end from the same T-Rex baseline.

• Context contract: precomputed RoboRAG-X token-level output, shape 32 x 1024, dtype float16, valid tokens 32.
• Fixed reference candidate: 004ac131050a539d91e6bdd4.
• Reference dropout during training: 0.10.
• Robot input: three RGB cameras on a 448 x 384 composed canvas.
• Action/proprioception contract: T-Rex 62-D bimanual action space.
• Action chunk: 32 steps.

Default inference lifecycle: Short-24

The bundled runtime defaults to Short-24 measured in robot control steps: reference conditioning is active for rollout steps [0,24) and is disabled from step 24 onward. The T-Rex client executes 16 actions before replanning, so the server uses one reference forward for steps 0-15, then performs one reference and one no-reference forward at the 16-31 block and splices exactly at local index 8. All subsequent forwards are no-reference. Thus this is 24 executed control steps, not 24 model calls and not a rounded 32-step window.

Every rollout must reset the server clock. The bundled client patch sends a unique episode_id, a reset request, the exact control_step, and execute_steps=16; this prevents lifecycle state from leaking between robot episodes. The server response includes a reference_lifecycle audit record for every chunk.

This is an inference-lifecycle setting. The published checkpoint itself was trained with the same fixed reference available per sample and reference dropout 0.10; it was not retrained with a 24-step temporal gate. Report Short-24 real-robot results separately from persistent-reference results.

The reference assignment, extraction receipt, precomputed Context Tokens, and audited MP4 are included under reference/.

• Human reference video
• Precomputed feature manifest
• Fixed task text: Pick up the fruit on the left side using your left hand and place it in the basket. Then, pick up the fruit on the right side using your right hand and place it in the basket.
• Runtime conditioning loads reference/context_tokens.safetensors directly.
• The MP4 is audit/visualization material and is not decoded during policy inference.
• The original embedding model (roborag-x-v5-four-domain-4222 at 2814bed03253a1b618da4722d3d23baf906a75ab1237a4626c828658abbcd89e) is provenance only: it is not downloaded, imported, or executed at inference time.

HF-only real-robot policy server

The repository contains the exact FastWAM source snapshot, model topology, Wan VAE, normalization statistics, fixed task embedding, precomputed Context Tokens, and a T-Rex-compatible ZMQ server. No GitHub checkout, training dataset, RoboRAG-X checkout, or embedding-model checkpoint is needed.

huggingface-cli download YunzeLiu/fastwam-trex62-prefilling-icl --local-dir fastwam_trex62
cd fastwam_trex62
python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r inference/requirements.txt
python -m pip install -e runtime --no-deps
python inference/serve_trex62.py --model-dir . --device cuda:0 --port 5678

Apply the bundled lifecycle patch to the matching T-Rex checkout, then copy the values in inference/trex_client_override.yaml into the site-specific hardware configuration:

patch -d /path/to/T-Rex -p1 < inference/trex_short24_client.patch

The critical settings are chunk_size: 32, execute_steps_per_chunk: 16, command_hz: 30, head_crop_box: null, three uncropped RGB cameras at 640x360, and use_tactile_refine: false. If the client cannot be patched, run python inference/reset_trex62_episode.py immediately before every rollout; without a per-rollout reset, Short-24 state would carry across episodes.

The server accepts the existing T-Rex pickle/ZMQ keys (image_head, image_wrist_left, image_wrist_right, state_slow, task_description) and the lifecycle keys (episode_id, control_step, execute_steps), and returns a 32x62 float32 action chunk. A {mode: "reset", episode_id: ...} request starts a fresh rollout. Per arm, the 62-D layout is local delta xyz (3), local delta rot6d (6), and absolute Sharpa hand targets (22): [left 31 | right 31]. It deliberately does not turn the deltas back into absolute EEF poses; the T-Rex client resolves them through differential IK.

An offline hardware-I/O check is also included:

python inference/predict_trex62.py --model-dir .   --head head.png --left-wrist left.png --right-wrist right.png   --state current_state_62d.npy --control-step 0 --execute-steps 16   --output actions.npy

Final offline validation

The original training gate is preserved verbatim at artifacts/convergence_gate.json. A passing gate is not a physical-robot success rate. For the side-branch release the gate is intentionally reported as failed rather than overwritten: its checkpoint is complete and the branch is live, but the legacy first/last-eight train-loss test is noisy at the already-converged base loss floor.

Reproduction

The matching code is yunzeliu/fastwam_icl at commit 91c068c6e347157b3779af78e0ea673d8b684aad. The resolved Hydra configuration, normalization statistics, complete metrics JSONL, and release manifest are bundled here.

Repository ID: YunzeLiu/fastwam-trex62-prefilling-icl.

Limitations and licensing

This model is specialized to the published camera/action/data contract and a single fixed human reference. Test first with execution disabled, then use the T-Rex collision, IK, smoothing, workspace, velocity, and emergency-stop protections. Never expose the pickle-based ZMQ endpoint to an untrusted network.

The FastWAM integration code is MIT licensed; base weights, third-party components, VLA2Vec, and EgoDex retain their own licenses and terms.

---

中文说明

本仓库包含完整的 prefilling T-Rex 62-D FastWAM-ICL 推理包：权重、固定人类参考视频、 预抽取的 32x1024 FP16 Context Tokens、任务文本 embedding、归一化统计、Wan VAE、 对应源码和 T-Rex ZMQ 推理服务均已包含在同一个 Hugging Face 仓库中。推理时直接 读取 reference/context_tokens.safetensors；不会解码参考视频，也不需要下载、导入或 运行 RoboRAG-X/embedding model。无需训练数据或 GitHub 代码即可加载。真机侧必须 使用完整的三路 640x360 RGB 图像，关闭默认 head crop 和 tactile refine，动作频率为 30 Hz，模型返回 32x62 的双臂 delta-EEF/绝对手指 关节目标。HF 推理默认采用严格的 Short-24：每个 rollout 的机器人控制步 0-23 使用 reference，第 24 步起关闭；16 步重规划的第二个 chunk 会在局部第 8 步做一次 reference/no-reference 双 forward 拼接。每个 rollout 必须通过随附的 client patch 或 reset_trex62_episode.py 重置服务端时钟。该时间门控是推理设置；现有 checkpoint 的 训练仍是固定 reference 加 0.10 dropout，并未按 24 步门控重新训练。首次上机必须 先断开执行器做 observation/action dry-run，并启用 T-Rex 自身的 IK、碰撞、限速和急停保护。