sandglasset-2spk-polyvox

PolyVOX Sandglasset 2-Speaker Separator

This repository contains the fine-tuned 2-speaker Sandglasset speech separation checkpoint used in the PolyVOX Samsung EnnovateX hackathon pipeline for real-time multi-user smart assistant command understanding in noisy smart-home environments.

Model Summary

• Task: 2-speaker single-channel speech separation.
• Sample rate: 16 kHz mono.
• Checkpoint: best_model.pt.
• Parameters: 4,928,770, under the 5M parameter hackathon budget.
• Architecture: compact Sandglasset-style encoder, bottleneck, gated TCN blocks, skip fusion, mask head, and transposed-convolution decoder.
• Intended pipeline role: separate overlapping speakers before ASR, command attribution, and SmartThings action routing.

Architecture

The model uses the project Sandglasset student architecture:

• Encoder: Conv1d(1, 256, kernel_size=16, stride=8) followed by ReLU.
• Bottleneck: GroupNorm(1, 256) and Conv1d(256, 128, 1).
• Separator: 24 gated Sandglass/TCN blocks with hidden size 384, kernel size 5, dilation cycle 1, 2, 4, 8, 16, and selected average-pooling downsampling.
• Mask head: Conv1d(128, 2 * 256, 1) with ReLU masks.
• Decoder: ConvTranspose1d(256, 1, kernel_size=16, stride=8).

The exact PyTorch implementation is included in modeling_sandglasset.py.

Training Data and Fine-Tuning

The checkpoint was trained with online dynamic mixing from LibriSpeech source utterances and MUSAN noise. The fine-tuning run focused on hard overlap and noise conditions for the real-time assistant setting.

Source run artifacts included in this repository:

• training_report.json
• training_report.md
• metrics.csv
• latest_summary.json

Raw LibriSpeech and MUSAN audio are not redistributed in this model repository.

Final KPI Evaluation

Final KPI report date: 2026-06-21.

Evaluation protocol: final sparse-overlap LibriMix-style KPI evaluation for the fine-tuned Sandglasset 2-speaker checkpoint.

Both final 2-speaker groups are far below the required 0.5 xRT runtime limit and the checkpoint satisfies the under-5M-parameter deployment constraint. The strict mean SI-SNR targets are not fully met for the 2-speaker final KPI groups; the model is strongest at low overlap and remains useful as a compact realtime separator.

Overlap Breakdown

Clean 2-speaker:

Noisy 2-speaker:

The same values are provided in machine-readable form in final_kpi_results.json.

Training Validation Metrics

Fine-tuned run: sandglass_overlap_noise_finetune_2spk_20260618_222414.

Quick Start

Install dependencies:

pip install -r requirements.txt

Separate a WAV file:

python inference.py input.wav --output-dir separated_outputs

Programmatic loading:

import torch
from modeling_sandglasset import load_sandglasset_checkpoint, separate_tensor

model, checkpoint = load_sandglasset_checkpoint("best_model.pt")
mixture = torch.randn(16000)
estimates = separate_tensor(model, mixture)
print(estimates.shape)  # [1, 2, time]

Intended Use

This model is intended for research and hackathon demonstration of real-time 2-speaker separation inside the PolyVOX smart assistant pipeline. It can be used to test overlapping command separation before ASR and downstream smart-home action routing.

Limitations

• Optimized for 16 kHz single-channel speech.
• Trained and validated on dynamic LibriSpeech-style speech mixtures with MUSAN noise augmentation, not every possible home acoustic condition.
• Speaker output order is permutation-invariant and should not be interpreted as a persistent speaker identity.
• Final KPI mean SI-SNR remains below the strict clean and noisy 2-speaker targets, especially in high-overlap mixtures.
• This package includes a PyTorch .pt checkpoint. Load it only from trusted sources, or create an additional Safetensors copy with convert_to_safetensors.py.

License and Attribution

The project code in this package is released under the MIT License. Training used LibriSpeech and MUSAN-derived data according to their respective dataset terms. See the parent PolyVOX repository documentation for full open-source and dataset attribution.