caip-roborag-x-four-domain-v1

CAIP RoboRAG-X four-domain v1

This repository contains the exact one-full-coverage-epoch CAIP checkpoint trained on the sealed four-domain RoboRAG-X data release: EgoDex, DROID, LIBERO, and RoboMIND-Franka-Dual. It is intended for controlled representation comparisons with RoboRAG-X and the upstream CAIP encoder.

Release identity

The loss values above are training diagnostics, not retrieval accuracy or a robot success rate.

Data and sampling contract

The underlying sealed train pool contains 1,695,684 windows. A covering epoch uses a deterministic weighted sampler with domain weights 20:10:35:35 (EgoDex:DROID:LIBERO:RoboMIND) and the following sample presentations:

• Image input: the sealed task-linked SAM3 interaction-object zoom at the current time, resized through CAIP's validation preprocessing.
• Text input: the sealed instruction used by CAIP's text-conditioned image pooler.
• Action target: strictly-future 4-second native action, resampled by physical time to 64 steps; native channels are prefix-copied and zero-padded to 378 dimensions.
• Normalization: per-domain train-only q01/q99 statistics inherited from the sealed RoboRAG-X release.
• Objective: upstream CAIP image-action SigLIP contrastive loss.

The included data_audit.json records decoded examples from every domain, source FPS/horizon, raw action width, padding checks, SAM3 score threshold, and hashes of the sealed runtime index and normalization data.

Files

• caip_roborag_x_four_domain_epoch1_inference.pt — model-only inference checkpoint.
• caip_roborag_x_four_domain_epoch1_training.pt — full OpenCLIP training checkpoint, including optimizer/scaler state.
• release_receipt.json — byte counts, SHA-256 values, tensor inventory, and terminal training record.
• config.yaml — formal one-epoch experiment contract.
• data_audit.json — real-data shape, action, image, sampler, and provenance audit.

Representative audited action-module tensor shapes include module.action_encoder.cls_token: [1, 1, 1024], module.action_encoder.input_proj.bias: [1024], module.action_encoder.input_proj.weight: [1024, 378], module.action_encoder.norm.bias: [1024], module.action_encoder.norm.weight: [1024], module.action_encoder.output_proj.bias: [1024], module.action_encoder.output_proj.weight: [1024, 1024], module.action_encoder.pos_embed: [1, 65, 1024].

Loading and reproduction

Use the CAIP/OpenCLIP implementation in yunzeliu/VLA2Vec at commit f3d8691a2bdbcae06e28bd33c79df6cdb861f078. The files are native PyTorch/OpenCLIP checkpoints, not a Transformers trust_remote_code package. Instantiate ViT-L-16-CrossAttn-256 with action shape 64 x 378, then load every key strictly. Do not silently use strict=False or omit the domain-specific preprocessing contract.

Limitations

• This is the requested one-epoch checkpoint; no claim of downstream-policy or physical-robot performance is made here.
• Cross-domain action coordinates do not share physical semantics merely because they occupy a common padded width. Always select the correct source-domain mapping and normalization.
• Query images assume a usable SAM3 interaction-object grounding. If grounding fails, resample a nearby current frame before embedding.
• Dataset, SigLIP2, and upstream CAIP terms still apply. Validate any policy use under the target robot's own safety stack.

Upstream CAIP

Architecture and objective follow yuvansharma/caip-vitl256 and the CAIP paper. This release changes the training data and action adapter inputs while keeping the original CAIP model topology.