octo-base-1.5

Octo Base

See https://github.com/octo-models/octo for instructions for using this model.

Octo Base is trained with a window size of 2, predicting 7-dimensional actions 4 steps into the future using a diffusion policy. The model is a Transformer with 93M parameters (equivalent to a ViT-B). Images are tokenized by preprocessing with a lightweight convolutional encoder, then grouped into 16x16 patches. Language is tokenized by applying the T5 tokenizer, and then applying the T5-Base language encoder.

Observations and tasks conform to the following spec:

Observations:

{
    image_primary: ('batch', 'history_window', 256, 256, 3),
    image_wrist: ('batch', 'history_window', 128, 128, 3),
}

Tasks:

{
    image_primary: ('batch', 256, 256, 3),
    image_wrist: ('batch', 128, 128, 3),
    language_instruction: {
        attention_mask: ('batch', 16),
        input_ids: ('batch', 16),
    },
}

At inference, you may pass in any subset of these observation and task keys, with a history window up to 2 timesteps.

This model was trained on a mix of datasets from the Open X-Embodiment dataset.

Updates for Version 1.5

• Language task tokens are now repeated at every timestep in the context window.
• Augmented the language instructions in the data with rephrasings from GPT-3.5.
• Bug fixes:
  • Turned off dropout in the diffusion head due to incompatibility with layer norm.
  • Fixed an off-by-one error with the attention mask.
  • Fixed an issue where different image augmentations did not get fresh random seeds.