fusion-embedding-2-tremor

fusion-embedding-2-tremor

<p align="center"> <img src="assets/tremor-banner.png" alt="Tremor — the motion sense for Fusion Embedding 2 (2B-Preview), Eximius Labs" width="100%"> </p> <div align="center">

</div>

Tremor is the inertial (IMU) sensor pack for Eximius Labs' fusion-embedding stack. It maps a short window of body-worn motion (a 3-axis accelerometer stream) into the Qwen3-VL-Embedding-2B text embedding space, so a robot's or wearable's motion history becomes searchable in plain language: "find when it was walking", "find when it picked something up".

Tremor is part of the fusion-embedding family and plugs into the same shared space as fusion-embedding-2. It targets the frozen Qwen3-VL-Embedding-2B text embeddings that the family is built on, and because fusion-embedding-2's text, image, and video paths are byte-identical to that base, Tremor's motion embeddings are directly comparable to that model's audio, text, image, and video in one 2048-d space. Motion becomes a first-class, language-addressable modality alongside the rest.

GitHub | fusion-embedding-2 | Technical report: arXiv:2607.18666

The family. Each sense is a separately loadable pack over the same frozen base: Tactus reads touch from a 32x32 pressure glove, Tactus Mat reads a 64x32 body pressure mat, Ember reads heat, and Tremor reads motion, with a Unitree-G1 head. Because the base is never modified, adding a sense costs a small trained head and an afternoon of compute rather than a new foundation model.

Model Overview

<p align="center"> <img src="assets/tremor_model_overview.png" alt="Tremor architecture: a frozen UniMTS encoder feeds a trained projector into the fusion-embedding shared space, where motion becomes searchable in natural language alongside every other modality" width="880px"> </p>

Tremor is a frozen UniMTS motion encoder plus a small trained projector. A single accelerometer window is placed at one joint of UniMTS's orientation-invariant skeleton-graph encoder (ST-GCN), which produces a 512-d motion feature; a projector of roughly 2.6M parameters maps that into the frozen Qwen base's 2048-d text space. Nothing is trained except the projector. Motion and text embeddings are L2-normalized and compared by cosine similarity.

The design choice that matters is the encoder: rather than train an IMU encoder from scratch, Tremor reuses a frozen, orientation-invariant pretrained one and learns only the projection into the language space. The pretraining prior, not our training, is what carries cross-subject and cross-mounting generalization.

See it in action

A full activity sequence, read live. A Unitree G1 performs each activity in simulation — sitting, walking, running, climbing — while a window slides across a real, held-out body-worn accelerometer stream (RealWorld, out of training) and Tremor's scores update in real time, tracking the motion as it changes. The robot shows each activity full-body; the signal and the predictions are real, frame-synced data.

<p align="center"> <img src="assets/tremor_robot_stream.gif" alt="A Unitree G1 performs sitting, walking, running and climbing full-body while Tremor's live scores track the activity on a real held-out accelerometer stream" width="860px"> </p>

On the robot's own IMU and camera. On a real Unitree G1 humanoid (Humanoid-Everyday), Tremor reads the robot's own IMU as language, live and frame-synced to its egocentric camera — zero simulation. As it switches from adjusting a phone stand to walking to a door, its IMU signature changes from flat to dynamic and Tremor's read tracks the activity in real time.

<p align="center"> <img src="assets/tremor_robot_demo.gif" alt="Live demo on a real Unitree G1: the robot's own egocentric camera with its frame-synced IMU (zero simulation); Tremor's read tracks the activity, from adjusting a phone stand to walking to a door" width="840px"> </p>

Any body-worn sensor. No robot is required — the same works on a wristband or wearable. Here the raw accelerometer stream slides by directly and Tremor's scores track sitting → walking → running → climbing on the held-out RealWorld set.

<p align="center"> <img src="assets/tremor_demo.gif" alt="Live classification on a held-out human-activity stream: a window slides across the accelerometer signal and Tremor's activity scores update in real time" width="720px"> </p>

Text → motion retrieval. Query an activity in plain language and Tremor returns the nearest motion windows. Below, on the RealWorld set held out of training entirely (zero-shot), each thumbnail is a retrieved window's 3-axis accelerometer trace; correct retrievals (the window's true activity matches the query) are outlined in green.

<p align="center"> <img src="assets/tremor_retrieval_gallery.png" alt="Text to motion retrieval on held-out RealWorld: each query returns its top-5 nearest accelerometer windows, correct retrievals outlined green" width="840px"> </p>

Training and Evaluation

Tremor is trained with InfoNCE against the frozen base's native chat-template text embeddings, on a diverse 16-source pool of real inertial data: 13 standard human-activity-recognition datasets, Ego4D (egocentric human), Humanoid-Everyday (Unitree robot), and Capture24 (151 subjects of free-living wrist accelerometry). Only the projector learns.

The single most important lesson of the project is the evaluation metric. Instance retrieval (finding the exact paired window) is misleading for this task and makes a working activity recognizer look dead. The right metric is k-way activity classification on held-out subjects and held-out datasets, and all numbers below use it (chance = 1/k).

Diversity is what makes the base general

Trained on one source (Ego4D alone), an IMU encoder generalizes poorly to sensor streams that look nothing like it. Tremor's headline result is zero-shot transfer to datasets held out of training entirely. On five held-out datasets, a single-source baseline (the identical architecture trained on Ego4D only) sits at or below chance, while Tremor generalizes.

<p align="center"> <img src="assets/tremor_v01_results.png" alt="Trained on 16 IMU sources, Tremor generalizes zero-shot to datasets held out of training entirely, far exceeding a single-source baseline that sits at or below chance" width="820px"> </p>

These are the single-sensor numbers the shipped inference.py produces: one accelerometer window in m/s^2, resampled from the sensor's rate to 20 Hz and placed at one fixed skeleton joint. That default joint is selected on in-domain held-out data disjoint from the four zero-shot sets above, so these numbers are not tuned on the benchmark. Feeding the model the sensor's true sample rate and unit convention is what recovers this accuracy; a rate-agnostic window scores well below it. (A research configuration that uses each dataset's own multi-sensor body placement reaches 0.545 mean, but that requires per-dataset sensor-mount knowledge and is not available to a general single-IMU input.)

On the two in-domain holds (held-out subjects/episodes of sources seen in training), Tremor reaches 0.40 5-way on both Ego4D and the Humanoid-Everyday robot set. The held-out datasets above are excluded from training entirely, so those numbers are genuine cross-dataset generalization.

Usage

<details> <summary>Requirements</summary>

• torch (CUDA recommended), numpy, scipy, transformers>=4.46, huggingface_hub
• The frozen UniMTS encoder code and weights (Apache-2.0 code):

  git clone https://github.com/xiyuanzh/UniMTS      # provides model.py::ST_GCN_18
  

  Point UNIMTS_REPO (or unimts_repo=) at the clone. The UniMTS weights download automatically from the hub.
• The frozen Qwen3-VL-Embedding-2B base downloads from its original repository.

</details> <details open> <summary>via <code>inference.py</code> (this repository)</summary>

import numpy as np
from inference import TremorEmbedder

tr = TremorEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-tremor",
    revision="v0.1-preview",
    unimts_repo="UniMTS",          # path to your git clone of xiyuanzh/UniMTS
)

# accel: a 3-axis accelerometer window in m/s^2, shape [3, T] (any length)
accel = np.load("window.npy")      # e.g. [3, 300]

# rank candidate activities; pass your sensor's true sample rate (Hz) so the window is
# resampled to the model's 20 Hz. Pass unit="g" if your sensor reports g rather than m/s^2.
for text, score in tr.rank(accel, ["walking", "sitting", "running", "climbing stairs"],
                           sample_rate_hz=50):
    print(f"{score:+.3f}  {text}")

# or embed motion and text into the shared space directly
m = tr.embed_motion(accel, sample_rate_hz=50)    # 2048-d, L2-normalized
t = tr.embed_text(["walking", "sitting"])        # 2048-d each

Give the model the sensor's true sample rate and an accelerometer in m/s^2 (gravity ~9.8); these are what the held-out numbers above are measured with. Embedding quality is also sensitive to the base's chat-template formatting; inference.py applies it for you, so match candidate activities against motion through this API rather than embedding text yourself.

</details>

Deploy on RunPod

One-click deploy the endpoint from the RunPod Hub (serverless, scales to zero when idle). Once it is running, call it:

curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"motion": [[...], [...], [...]]}}'

motion is a 3-axis accelerometer window shaped [3, T]. Use text instead to embed an activity query. Returns 2048-d vectors, so motion and text land in one space. The Unitree G1 head deploys separately at the G1 hub page.

Related models

Tremor comes in two tiers, both built on fusion-embedding-2:

Start with this general base — it is the tier that generalizes to sensors it has never seen. Move to the per-fleet Tremor-G1 head only when you deploy on that platform and want maximum in-domain accuracy. Specializing a head raises in-domain accuracy but does not improve cross-dataset transfer; the two do different jobs. Deployment guidance (sensor contract, streaming, on-device inference, per-fleet fine-tuning) is in the Tremor-G1 card.

The same shared space also has a touch sense: fusion-embedding-2-tactus embeds 32x32 pressure/taxel arrays (FSR gloves, e-skins, robot hands) and matches to exceeds the STAG (Nature 2019) supervised baseline while remaining open-vocabulary.

Engram

This pack is one of the modalities Engram searches. Engram is the open cross-modal memory layer for physical AI: it indexes a robot's video, audio, and motion into one embedding space and answers questions about it in plain language, including temporal reasoning that retrieval alone cannot.

pip install engram-robomem

Repo: https://github.com/Eximius-Labs/engram · PyPI: https://pypi.org/project/engram-robomem · Playground: https://www.eximiuslabs.com/playground

License

The trained projector weights in this repository are released under CC-BY-NC-4.0 (non-commercial).

This posture is deliberate and reflects the model's lineage. Tremor's frozen encoder is UniMTS, which was pretrained on synthetic IMU derived from the AMASS / HumanML3D motion corpora. AMASS is released under a non-commercial research license, so we ship Tremor non-commercial to respect that lineage. A commercially-clean encoder (re-pretrained on permissively licensed motion) is future work; a commercial license may follow.

Limitations

• Accelerometer only. The released UniMTS encoder is accelerometer-only; gyroscope is not used. Rotational-rate information is not currently modeled.
• Cross-dataset ceiling. Zero-shot cross-dataset accuracy plateaus around 0.50 (5-way). This is a research preview, not a production activity classifier. The right use is language-addressable motion search, not high-stakes classification.
• Single-sensor, single-joint. One IMU mapped to one skeleton joint, matching a fixed-mount deployment (a robot torso, a wristband). Multi-sensor fusion is not modeled.
• English text only, and the base's chat-template formatting must be used.
• Specialization vs. generalization. Fine-tuning on a specific platform's data raises accuracy on that platform substantially but does not improve cross-dataset transfer. Tremor is the general base; per-fleet fine-tuning is a separate recipe.

Citation

If you use Tremor, please cite this repository and acknowledge the frozen components it builds on:

@misc{tremor2026,
  title  = {Tremor: an inertial-motion sensor pack for the fusion-embedding space},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  note   = {Eximius Labs. Model weights CC-BY-NC-4.0.},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor}
}

Tremor's frozen motion encoder is UniMTS (Zhang et al., UniMTS: Unified Pre-training for Motion Time Series, NeurIPS 2024), pretrained on synthetic IMU derived from AMASS (Mahmood et al., ICCV 2019). The text space is Qwen3-VL-Embedding-2B. Please cite those works as well.