fusion-embedding-2-tactus

fusion-embedding-2-tactus

<p align="center"> <img src="assets/tactus-banner.png" alt="Tactus, the tactile sense for Fusion Embedding 2 (2B-Preview), Eximius Labs" width="100%"> </p> <div align="center">

</div>

Tactus is the tactile sensor pack for Eximius Labs' fusion-embedding stack. It maps a short window of pressure-array frames (a 32x32 taxel grid, the signal class produced by resistive/FSR gloves, e-skins, and instrumented robot hands) into the Qwen3-VL-Embedding-2B text embedding space, so touch becomes searchable in plain language: recognition is a text query, not a trained classifier head.

Tactus reads low-dimensional pressure arrays, not optical tactile images. Optical sensors (GelSight, DIGIT) already have strong models (TVL, UniTouch, Sparsh); the cheap, widely-shipped resistive arrays did not. To our knowledge Tactus is the first open model to put this sensor class in a text-aligned, cross-modal embedding space.

Tactus is part of the fusion-embedding family alongside Tactus Mat (the same pack trained for a 64x32 body pressure mat), Tremor (motion) and Ember (thermal). Its embeddings target the canonical readout of fusion-embedding-2, so tactile windows are directly comparable to that model's text, image, video, and audio in one 2048-d space, and drop into the Engram memory layer (pip install engram-robomem) as a first-class sense.

GitHub | fusion-embedding-2 | Live playground | Family report (arXiv:2607.18666) | Tactus report (arXiv:2608.04043)

Model Overview

<p align="center"> <img src="assets/tactus_model_overview.png" alt="Tactus architecture: calibrated pressure windows pass through an MAE-pretrained per-frame trunk, a learned frame fusion, and a trained projector into the fusion-embedding shared space, where touch becomes searchable in natural language alongside every other modality" width="880px"> </p>

Tactus is a trained CNN trunk plus projector over pressure windows. Each 32x32 frame passes through a ResNet-18-width trunk (3x3 stem, four stages; 32x32 -> 4x4 spatial map); the K frames of a grasp window are fused by a learned 1x1 convolution over their concatenated feature maps, pooled, and projected into the frozen base's 2048-d text space. The trunk is initialized by masked-autoencoder pretraining (mask 0.6, per-patch normalized targets) on 144k unlabeled STAG-family pressure frames, then fine-tuned contrastively against canonical text embeddings of natural grasp phrases.

The design choice that matters is the data path: pressure is normalized with the sensor's own calibration affine (clip((raw - 500) / 150, 0, 1), the STAG reference preprocessing), and pretraining stays same-sensor. In our ablations, correct normalization and same-sensor MAE were worth more than every architecture change combined, while cross-sensor pretraining pooling gave nothing, consistent with published findings (HTT, TacVerse).

See it in action

Real held-out grasps, recognized from pressure alone. Each panel is a genuine STAG test frame (the most active frame of that class in the held-out split, by total pressure) with the text query the model matches it against: no camera, no trained classifier head. Across the full test split the model averages 0.77 top-1 and 0.94 top-3 over 27 such queries.

<p align="center"> <img src="assets/tactus_test_grasps.png" alt="Four real held-out STAG test pressure maps with their text queries: a mug, scissors, a full can, safety glasses, each recognized from the 32x32 pressure pattern alone" width="860px"> </p>

Training and Evaluation

Tactus trains in two stages on the STAG datasets (Sundaram et al., Nature 2019): a masked-autoencoder pretrain over every STAG-family pressure frame (classification + blindfolded + weights + handposes, 144k frames including unlabeled ones, supervised test frames excluded), then contrastive fine-tuning of the whole head against canonical text embeddings of grasp phrases, with STAG-style cluster sampling (each training window draws diverse frames from across a recording rather than consecutive near-duplicates).

Evaluation is 27-way object recognition on fully held-out test recordings, scored as cosine ranking against text queries (open-vocabulary; the model never trains a classifier head).

Interpreting these numbers: the recipe's mean exceeds the original paper's supervised CNN while performing a harder task (open-vocabulary text queries against a frozen language space, versus a 27-way trained classifier), though by less than one standard error; we describe the result as matching to exceeding the original baseline, with best runs at 0.83, rather than claiming a definitive margin. Top-3 accuracy is stable across every run. Our evaluation mirrors STAG's cluster-sampling test protocol but is not their byte-identical harness. Same-sensor MAE pretraining accounts for about +7 points over training from scratch. Full recipe, ablations, and negative results: results.json and the GitHub repository.

Usage

<details> <summary>Requirements</summary>

• torch (CUDA recommended), numpy, safetensors
• pip install fusion-embedding[hf] for the text side (the canonical whitened readout Tactus was trained against; embedding text any other way will misrank)
• The frozen base downloads from EximiusLabs/fusion-embedding-2-2b-preview.

</details> <details open> <summary>via <code>inference.py</code> (this repository)</summary>

import numpy as np
from inference import TactusEmbedder

ta = TactusEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-tactus",
                                    revision="v0.1-preview")

# a grasp window: [F, 32, 32] pressure frames (uint8 0-255 or floats in [0, 1]);
# for raw sensor counts pass raw="stag" to apply the calibration affine
window = np.load("grasp.npy")

for text, score in ta.rank(window, ["a mug", "scissors", "a full soda can", "an empty hand"]):
    print(f"{score:+.3f}  {text}")

# or embed both sides into the shared space directly
p = ta.embed_pressure(window)          # 2048-d, L2-normalized
t = ta.embed_text(["holding a mug"])   # canonical text embedding, same space

Pressure embeddings land in the same space as fusion-embedding-2's text, image, video, and audio, and as Tremor's motion, so cross-modal search over a robot session works out of the box through Engram. Match text against pressure through this API rather than embedding text with the raw base model; Tactus was trained against the canonical whitened readout, and other text paths will misrank.

</details>

Related models

Tactus joins the fusion-embedding sense packs, all built on fusion-embedding-2:

All packs embed into one 2048-d space, so a query can match across senses. The Engram memory layer (pip install engram-robomem) wires them into a searchable robot session memory with temporal reasoning.

License

The trained weights in this repository are released under CC-BY-NC-4.0 (non-commercial).

This reflects the training data's lineage: Tactus is trained on the STAG datasets, which are released for non-commercial research use. A commercially-clean retrain (on permissively licensed pressure corpora) is future work; a commercial license may follow.

Limitations

• Run-to-run variance. The training recipe's top-1 varies +/-0.06 across seeds (0.70-0.83 over four runs). The released checkpoint is a strong draw, and the mean is reported alongside it. Seed stabilization is active work.
• One sensor family. Trained on one glove (STAG's 32x32 grid, 548 taxels). Our cross-sensor experiments show transfer to other taxel geometries needs fine-tuning, not zero-shot use; the input path accepts any [F,32,32] window, and other resolutions must be resampled.
• 27-object vocabulary at eval. Open-vocabulary means text queries, not tested open-set generalization to arbitrary unseen object categories; treat novel-category recognition as unvalidated.
• Research preview. Not a production classifier. The intended use is language-addressable touch inside a multimodal memory, not high-stakes recognition.
• English text only, through the canonical readout (fusion-embedding package); do not embed text with the raw base model.

Citation

If you use Tactus, please cite this repository and the dataset it builds on:

@article{tactus2026,
  title   = {Tactus: Open-Vocabulary Object Recognition from Low-Cost
             Pressure Arrays},
  author  = {Tonmoy, Abdul Basit},
  journal = {arXiv preprint arXiv:2608.04043},
  year    = {2026}
}

Tactus trains on STAG (Sundaram et al., Learning the signatures of the human grasp using a scalable tactile glove, Nature 2019); please cite that work when using the benchmark numbers. The text space is Qwen3-VL-Embedding-2B.