EgoNoiseSeparation

EgoNoiseSeparation

Open-Set Ego-Noise Separation for Legged-Robot Audition via Annotation-Free Adaptation and Pretrained-Model Transfer

Koki Shoda, Jun Younes Louhi Kasahara, Aoba Koyanagi, Qi An, and Atsushi Yamashita (The University of Tokyo).

This repository provides the G1 and Go1 Transfer-DiT checkpoints trained with RecurGraph-selected ego-noise-dominant clips for open-set ego-noise separation in legged-robot audition.

• Code: https://github.com/kokieto/EgoNoiseSeparation
• Project page: https://kokieto.github.io/EgoNoiseSeparation/

Checkpoints

Each file is a training checkpoint containing the model state and training metadata. SHA256 checksums are provided in SHA256SUMS.

Setup

SAM-Audio code and base model weights are not included in this repository. Install SAM-Audio and download the base model before using a Transfer-DiT checkpoint. Access to the base model may require accepting its access terms on Hugging Face.

git clone https://github.com/kokieto/EgoNoiseSeparation.git
cd EgoNoiseSeparation
conda create -n l2hww python=3.11 -y
conda activate l2hww
python -m pip install -e .
bash scripts/setup_sam_audio.sh
huggingface-cli login
python scripts/download_sam_audio_weights.py --model-id facebook/sam-audio-small

Inference

Download the checkpoint for the target robot from the table above, then run the inference script in the activated conda environment:

python scripts/separate_audio.py \
  --audio path/to/mixed.wav \
  --checkpoint path/to/best.pt \
  --out-dir outputs/demo

Transfer-DiT ignores text, video, and span prompts internally. The optional --description argument is retained for SAM-Audio processor batching.

License

The released checkpoints are provided under the Creative Commons Attribution 4.0 International License. SAM-Audio code and base weights are third-party assets with their own license terms:

• SAM-Audio code: https://github.com/facebookresearch/sam-audio
• SAM-Audio small weights: https://huggingface.co/facebook/sam-audio-small