lbind-audio-vision

LBind: Multi-Modal Embeddings

Model Details

Model Description

LBind is a multi-modal embedding model that supports image, video, audio, text, and 3D point cloud inputs. All modalities are projected into a shared embedding space, enabling cross-modal similarity computations. The model builds on top of three other models; Perception Encoder, ImageBind, and Uni3D. As indicated by the figure in the top, data is first embedded individually by the three said models. Audio and 3D point cloud embeddings are successively projected with an MLP into the embedding space of the Perception Encoder. The model produces unit-norm embeddings directly usable for similarity comparisons via dot-products ([cosine similarity]).

This version loads the audio and vision encoders. If you would like the version that loads all encoders, please refer to lbind-full, or if you would like the version that loads 3D points, please refer to the 3D-points-vision model.

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

The model is intended to be used with direct file-inputs of the said modalities; image, video, audio, 3D, and text. It will produce a 1024 dimension embedding per input, suited for similarity computations.

Downstream Use

The model could be used to build multimodal LLMs, generative models, and systems that perceive their surroundings via both visual, and audio.

Bias, Risks, and Limitations

The model was built on data specified in the paper. As such, it will be biased towards data that "lives on the internet." For specific use-cases, a subsequent fine-tuning stage may be necessary.

How to Get Started with the Model

Option 1
If you want to work within the repository, use uv to install the necessary dependencies.

git clone https://github.com/strictnullchecks/lbind
cd lbind
uv sync

Option 2
You can also install it as an external dependency for another project:

# Option 2.a
python -m pip install git@https://github.com/strictnullchecks/lbind
# Option 2.b; or install a local, editable version
git clone https://github.com/strictnullchecks/lbind
cd /path/to/your/project
python -m pip install -e /path/to/lbind

[!WARNING] If you are running a project with pytorch=2.8.0, you should install torchcodec=0.7.0 (as opposed to the =0.8.0) which is automatically installed with uv. torchcodec=0.8.* matches pytorch=2.9.0.

[!NOTE]
The 3D point cloud backbone has a few custom CUDA kernels that you might want to compile. To do that, you will have to do use Option 1 or Option 2.b above to get a local copy of the repository and compile the kernels.

Loading the Model

import torch
from lbind import LBindModel, LBindProcessor

model = LBindModel.from_pretrained("strictnullchecks/lbind-full")
processor = LBindProcessor.from_pretrained("strictnullchecks/lbind-full")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()
processor = processor.to(device)

Processing Multi-Modal Inputs

inputs = {
    "image": ["examples/dog.png", "examples/cat.png"],
    "video": ["examples/dog.mp4", "examples/cat.mp4"],
    "audio": ["examples/dog.mp4", "examples/cat.mp4"],
    "text": ["A dog is howling in the street", "A cat is sleeping on the couch"],
}

with torch.inference_mode():
    batch = processor(inputs, return_tensors="pt")  # set text_file_paths=True if passing text file paths instead of strings
    outputs = model.forward(**batch)

Computing Cross-Modal Similarities

keys = list(outputs.keys())
for i, modality in enumerate(keys):
    for j, modality2 in enumerate(keys[i + 1:]):
        result = outputs[modality] @ outputs[modality2].T
        print(f"{modality} x {modality2}:")
        print(result.cpu().detach().numpy())
        print('='*26)

Expected Output:

image x video similarity:
[[0.48 0.42]
 [0.41 0.6 ]]
==========================
image x audio similarity:
[[0.07 0.05]
 [0.02 0.12]]
==========================
image x text similarity:
[[0.16 0.07]
 [0.08 0.14]]
==========================
video x audio similarity:
[[0.19 0.08]
 [0.03 0.16]]
==========================
video x text similarity:
[[0.26 0.05]
 [0.11 0.14]]
==========================
audio x text similarity:
[[ 0.12 -0.  ]
 [ 0.07  0.09]]
==========================

Note: The image/video similarity is significantly higher because they share the same vision encoder.

Evaluation

We have evaluated the model on multiple benchmarks. We highlight that LBind is performing close to as well as models 4 and 17 times larger. Please see more information on performance benchmarks on the lbind-full model card.