audio-scene-synthesis

Audio Scene Synthesis — trained checkpoints

Checkpoints for Sukhvansh2004/Audio-Scene-Synthesis, a refined Audio-LDM for text-to-audio generation that replaces AudioLDM's conditioning stack with Microsoft CLAP (HTSAT audio encoder + GPT-2 text encoder), plus several side experiments.

Almost everything here is weights we trained ourselves, plus a handful of generated samples. The one exception is third-party/clap_htsat_tiny.pt, which is mirrored because no upstream release ships it. The other third-party components — the pretrained mel VAE, HiFi-GAN vocoders, AudioMAE, the official AudioLDM checkpoints — are not mirrored here; the repository's scripts/setup_data.sh fetches those from their original sources.

Files

audioldm/ — the main model

The refined model reached a lower training loss at 70k steps than the baseline did at 500k.

dcase2023-task7/ — Foley sound synthesis experiment

Our fork of the DCASE 2023 Task 7 baseline, in experiments/dcase2023_task7/.

pixelsnail-melspec/ — autoregressive spectrogram model

clap-autoencoder/

third-party/ — mirrored, not ours

samples/ — generated audio, for a quick listen

Ten 16 kHz mono clips, 10.24 s each, so you can hear what the refined model produces without downloading a 5.9 GB checkpoint. samples/index.csv records the prompt, source split and provenance of each one.

• samples/curated/ (4) — a hand-picked demo set. The exact checkpoint that produced these was not recorded.
• samples/audiocaps-test/ (6) — from a validation pass at global_step 70005, the same training run as the 69,999 checkpoint above, on held-out AudioCaps test captions. Filenames are the prompts.

Usage

The repository's setup script places everything for you:

git clone https://github.com/Sukhvansh2004/Audio-Scene-Synthesis.git
cd Audio-Scene-Synthesis
./scripts/setup_data.sh --minimal      # pretrained components + the refined model
./scripts/setup_data.sh --experiments  # the DCASE and PixelSNAIL weights

Or fetch a single file directly:

from huggingface_hub import hf_hub_download

ckpt = hf_hub_download(
    "Sukhvansh/audio-scene-synthesis",
    "audioldm/checkpoint-fad-133.00-global_step=69999.ckpt",
)

Then run inference with the matching config:

python3 audioldm_train/infer.py \
  --config_yaml audioldm_train/config/2023_08_23_reproduce_audioldm/audioldm_custom.yaml \
  --list_inference tests/captionlist/inference_test.lst \
  --reload_from_ckpt "$ckpt"

Note the known CUDA driver issue with inference documented in the repository README.

Licence and attribution

Released under CC-BY-NC-4.0, matching the licence of the pretrained AudioLDM checkpoints these models were built against — non-commercial use only.

The work builds on:

• AudioLDM and AudioLDM-training-finetuning (MIT) — Liu et al., ICML 2023
• Microsoft CLAP (MIT)
• LAION CLAP, HiFi-GAN
• DCASE 2023 Task 7 baseline and liuxubo717/sound_generation

Training data: AudioCaps (Kim et al., NAACL 2019) over AudioSet audio. The DCASE models were trained on the DCASE 2023 Task 7 development set (Zenodo 8091972), part of which is BBC Sound Effects material provided for that challenge under research-use-only terms.

@article{liu2023audioldm,
  title={{AudioLDM}: Text-to-Audio Generation with Latent Diffusion Models},
  author={Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo
          and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D},
  journal={Proceedings of the International Conference on Machine Learning},
  year={2023}
}