robot-arm-voice-command-classifier

Robot Arm Voice Command Classifier

A small speech-classification project built to recognize five voice commands plus a no_command class from 3-second audio clips.

The final model uses a frozen HuBERT encoder (facebook/hubert-base-ls960) to extract frame-level speech representations, followed by an LSTM classifier for temporal command detection.

Commands

Dataset

The published dataset includes:

• original command recordings
• real non-command recordings
• robot movement noise
• DEMAND home-environment background noise used in the experiments
• generated 3-second synthetic clips
• frame-level labels used for training

The final synthetic dataset contains 10,400 clips:

• 8,000 training
• 1,200 validation
• 1,200 test

Audio is processed at 16 kHz with a fixed duration of 3 seconds.

Model

The model pipeline is:

3-second audio
      ↓
HuBERT Base
(frozen encoder)
      ↓
149 × 768 frame representations
      ↓
Layer Normalization
      ↓
LSTM (128 units)
      ↓
Dense layer
      ↓
6-class frame logits

HuBERT is not fine-tuned in this project. It is used as a frozen feature encoder while the downstream LSTM classifier is trained.

Final Trigger Configuration

The final deployment configuration was selected after temperature and class-specific threshold tuning.

• Temperature: 2.2
• False-trigger clips on the test set: 0
• Event recall: 85.33%
• Wrong-action clip rate: 1.42%
• Exact clip accuracy: 82.38%

Thresholds and minimum consecutive-frame requirements are different for each command and are documented in the training and inference notebooks.

Try the Model

Install the dependencies:

pip install -r requirements.txt

Open:

notebooks/04_single_audio_inference.ipynb

Then change:

AUDIO_PATH = Path("your_3_second_audio.wav")

The notebook will:

1. load the WAV file
2. convert it to mono
3. resample it to 16 kHz if needed
4. pad or trim it to 3 seconds
5. run HuBERT + LSTM inference
6. apply temperature scaling and command-specific trigger thresholds
7. print the detected command

The pretrained HuBERT files are downloaded automatically from Hugging Face the first time the model is loaded.

Notebooks

The notebooks follow the original experiment pipeline:

01_data_synthesis_and_split.ipynb
        ↓
02_hubert_frame_label_preprocessing.ipynb
        ↓
03_hubert_lstm_training_evaluation_and_tuning.ipynb
        ↓
04_single_audio_inference.ipynb

01 — Dataset Synthesis and Split

Creates reproducible source splits and generates the synthetic 3-second dataset.

02 — HuBERT Frame-Label Preprocessing

Creates frame-level labels and weights aligned with HuBERT's 149 output frames.

03 — HuBERT + LSTM Training, Evaluation, and Tuning

Contains the final V4 training run, test evaluation, temperature scaling, threshold tuning, and saved results.

04 — Single Audio Inference

Loads the trained model and runs inference on one WAV recording.

Repository Structure

robot-arm-voice-command-classifier/
├── README.md
├── requirements.txt
├── LICENSE
├── data/
│   ├── source_commands/
│   ├── source_backgrounds/
│   ├── source_splits/
│   ├── synthetic_trigger_data/
│   └── processed/
├── notebooks/
│   ├── 01_data_synthesis_and_split.ipynb
│   ├── 02_hubert_frame_label_preprocessing.ipynb
│   ├── 03_hubert_lstm_training_evaluation_and_tuning.ipynb
│   └── 04_single_audio_inference.ipynb
├── model/
│   └── best_model.weights.h5
└── results/
    ├── training/
    ├── test_forward_pass/
    ├── temperature_tuning/
    ├── threshold_tuning/
    └── evaluation/

Reproducibility Note

The training and evaluation notebooks preserve the outputs from the original experiment. The final training run was computationally expensive, so the notebooks were cleaned for publication without rerunning the training process.

Because the saved outputs come from the original run, some printed log paths may still show the original local project directories. The executable notebook code uses the public repository structure shown above.

Intended Use

This project is intended as a portfolio and research demo for small-vocabulary speech-command classification. It is not designed as a general-purpose speech-recognition system.