Downloads · 30 days
0
bpiyush/sound-of-water-models
sound-of-water-models is a audio classification model from bpiyush. Use it for the audio classification task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
In this folder, we provide the following trained model checkpoints:
Downloads · 30 days
0
Access
Public
Updated Jan 13, 2025
Repo size
764 MB
Likes
3
Public
Click a slice to open those files.
.pth756 MB · 99%
From the Hugging Face model README
In this folder, we provide the following trained model checkpoints:
<p align="center"> <img src="./assets/pitch_on_spectrogram-compressed.gif" alt="Teaser" width="100%"> </p>Key insight: As water is poured, the fundamental frequency that we hear changes predictably over time as a function of physical properties (e.g., container dimensions).
TL;DR: We present a method to infer physical properties of liquids from just the sound of pouring. We show in theory how pitch can be used to derive various physical properties such as container height, flow rate, etc. Then, we train a pitch detection network (wav2vec2) using simulated and real data. The resulting model can predict the physical properties of pouring liquids with high accuracy. The latent representations learned also encode information about liquid mass and container shape.
Arxiv link: https://arxiv.org/abs/2411.11222
Check out the demo here. You can upload a video of pouring and the model estimates pitch and physical properties.
First, install the repository from github.
git clone [email protected]:bpiyush/SoundOfWater.git
cd SoundOfWater
Then, install dependencies.
conda create -n sow python=3.8
conda activate sow
# Install desired torch version
# NOTE: change the version if you are using a different CUDA version
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
# Additional packages
pip install lightning==2.1.2
pip install timm==0.9.10
pip install pandas
pip install decord==0.6.0
pip install librosa==0.10.1
pip install einops==0.7.0
pip install ipywidgets jupyterlab seaborn
# if you find a package is missing, please install it with pip
Then, use this snippet to download the models:
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="bpiyush/sound-of-water-models",
local_dir="/path/to/download/",
)
To run our models on examples of pouring sounds, please see the playground notebook.
If you would like to use our dataset for a different task, please download it from here.
We provide audio models trained to detect pitch in the sound of pouring water. We train these models in two stages:
wav2vec2 on this data.Here, we provide checkpoints for both the stages.
<table style="font-size: 12px;" class="center"> <tr> <th><b> File name </b></th> <th><b> Description </b></th> <th><b> Size </b></th> </tr> <tr> <td><a href="https://huggingface.co/bpiyush/sound-of-water-models">dsr9mf13_ep100_step12423_synthetic_pretrained.pth</a></td> <td>Pre-trained on synthetic data</td> <td>361M</td> </tr> <tr> <td><a href="https://huggingface.co/bpiyush/sound-of-water-models">dsr9mf13_ep100_step12423_real_finetuned_with_cosupervision.pth</a></td> <td>Trained with visual co-supervision</td> <td>361M</td> </tr> </table> <!-- Add a citation -->If you find this repository useful, please consider giving a star ⭐ and citation
@article{sound_of_water_bagad,
title={The {S}ound of {W}ater: {I}nferring {P}hysical {P}roperties from {P}ouring {L}iquids},
author={Bagad, Piyush and Tapaswi, Makarand and Snoek, Cees G. M. and Zisserman, Andrew},
journal={arXiv},
year={2024}
}
@inproceedings{
bagad2024soundofwater,
title={The {S}ound of {W}ater: {I}nferring {P}hysical {P}roperties from {P}ouring {L}iquids},
author={Bagad, Piyush and Tapaswi, Makarand and Snoek, Cees G. M. and Zisserman, Andrew},
booktitle={ICASSP},
year={2025}
}
<!-- Add acknowledgements, license, etc. here. -->
We also want to highlight closely related work that could be of interest:
Our model is based on wav2vec2 which is trained on a large-scale speech recognition data. While this data is not as large as usual datasets in AI, it may still have undesirable biases that are present in the training data.