maestro-lstm-hybrid

two_stream_attn_v1_2layer_ld_cong_finetune_20260515T134706Z

A real-time hand gesture classifier trained on the LDJI hybrid gesture dataset (LD-ConGR swipes and fist, Jester zooms, IPN Hand pointing and background).

This model is part of the Maestro pipeline that enables touchless control of presentation and meeting software through hand gestures captured from a standard webcam using MediaPipe for landmark extraction.

Model Description

• Architecture: EnhancedTwoStreamLSTM (BiLSTM h=96×2, MHA 8 heads, proj=96, mean+max pool, MLP gate)
• Parameters: 1,208,554
• Input: (batch, 16, 147) — 16-frame sliding window at 30 FPS ≈ 533 ms
• Output: Softmax logits over 10 gesture classes
• Inference latency: < 1 ms per call (CPU, single sample)
• Feature schema: feature-schema-v5

Architecture

EnhancedTwoStreamLSTM splits the 147-dim feature vector into two parallel streams and processes them through a BiLSTM + self-attention + MLP-gate pipeline:

Input (B, T=16, 147)
    │
    ├─ Stream A — Pose/Shape (73 dims)
    │   Linear+LN+GELU → 96
    │   2-layer BiLSTM (h=96) → (B, T, 192)
    │   LayerNorm → Self-MHA (8 heads) + residual + post-LN
    │   mean+max pool → pool_LN → ctx_a (B, 192)
    │
    ├─ Stream B — Motion/Dynamics (74 dims)
    │   (identical structure) → ctx_b (B, 192)
    │
    ├─ MLP cross-stream gate
    │   gate_a = Sigmoid(
    │     Linear(96→192)(
    │       Tanh(Linear(192→96)(ctx_b))))
    │   ctx_a  = LN(ctx_a × gate_a + ctx_a)
    │   gate_b = Sigmoid(
    │     Linear(96→192)(
    │       Tanh(Linear(192→96)(ctx_a))))
    │   ctx_b  = LN(ctx_b × gate_b + ctx_b)
    │
    └─ cat(ctx_a, ctx_b) → (384,)
       LN → Linear(384→192) → GELU → Dropout → Linear(192→10)

Design rationale:

• BiLSTMs encode temporal order via their recurrent cell state — no positional encoding needed.
• Mean+Max pooling captures both sustained gesture shape (mean) and transient click events (max).
• The 2-layer MLP gate provides non-linear cross-modal recalibration at ~37 K params (vs ~263 K for full MHA cross-attention with a degenerate mean-pooled query).

Gesture Classes

Gesture Usage In Presentation System

Feature Schema (feature-schema-v5)

How to Use

import torch
from huggingface_hub import hf_hub_download
from maestro.infrastructure.model.checkpoint_loader import load_inference_artifact

# Download the artifact (cached after first call)
local_path = hf_hub_download(
    repo_id="ntsrigaud/maestro-lstm-hybrid",
    filename="two_stream_attn_v1_2layer_ld_cong_finetune_20260515T134706Z_inference.pt",
)

# Load the artifact (includes model, class labels, and feature schema)
artifact = load_inference_artifact(
    artifact_path=local_path,
    device=torch.device("cpu"),
)
artifact.model.eval()

# Build a 147-dim feature vector using LandmarkFeatureTransformer
# and fill a 16-frame SlidingWindowSequenceBuffer, then:
with torch.no_grad():
    # tensor shape: (batch=1, T=16, F=147)
    window_tensor = torch.tensor(window_np, dtype=torch.float32).unsqueeze(0)
    logits = artifact.model(window_tensor)
    pred_class = artifact.class_labels[logits.argmax(dim=1).item()]

Training Dataset

• Source: LDJI corpus of 28,882 windows pooled from LD-ConGR (fist and the four swipes), Jester (two zoom classes, center-cropped to 16 frames) and IPN Hand (point_one, point_two, unknown)
• Used classes: 10 (9 active gestures + unknown background)
• Dataset split: 70% train / 15% val / 15% test (stratified by class)
• Augmentation: temporal scale ±20%, spatial jitter σ=0.005

Training Strategy

Two-phase transfer learning pipeline:

• Phase 1 (pretraining): backbone pretrained on Jester.
• Phase 2 (fine-tuning): classification head replaced and model fine-tuned on the 10-class LDJI vocabulary.
• Stage A (frozen backbone): 10 epochs of head-only warmup.
• Stage B (full model): up to 80 epochs of joint fine-tuning (early stopping selected epoch 46) with a 5-epoch backbone warmup.

Training Configuration

Evaluation Results (Test Set)

Per-Class Recall

Model Versions

Both share the same architecture (1,208,554 parameters, 16-frame window) and the LDJI training corpus. The reference model scores slightly higher offline but one swipe gesture was less reliable in live testing, so the production model remains the default. Other artifacts in this repository are earlier experiments.

Comparison with Previous Architecture

License

These weights are for academic and non-commercial research use only (see MODEL_LICENSE.md), because they were trained on LD-ConGR (academic research only), Jester (research-use license) and IPN Hand (CC BY 4.0). The Maestro source code is Apache-2.0.

Limitations and Risks

• Trained on a hybrid of Jester, IPN Hand and LD-ConGR, Jester and IPN Hand recordings. Performance may degrade with unusual hand sizes, skin tones, or lighting conditions not represented in training data.
• The unknown class represents background/transition frames. At runtime, predictions are filtered through per-class confidence thresholds defined in production_hybrid.yaml.
• Requires mediapipe>=0.10.14 for landmark extraction at inference time.
• Not intended for safety-critical or accessibility-critical applications.
• Performance was measured on a held-out test split from the same dataset; real-world generalisation may differ.

Environmental Impact

Training was performed on CPU/MPS. Estimated training time: ~10 minutes. Estimated CO₂ equivalent: negligible (<0.001 kg CO₂eq).

---

Generated by the Maestro training pipeline on 2026-05-13.