wav2vec2bert_2_diffusion

Model Card for Model ID

This is a Wav2Vec2-BERT model that can be used as an audio conditioning mechanism for Stable Diffusion instead of the CLIP text encoder.

Model Details

Model Description

• Developed by: Youssef Kardous
• Model type: Enhanced Wav2Vec2-BERT with a custom loss function and adapter
• License: apache-2.0
• Finetuned from model : facebook/w2v-bert-2.0

Model Sources

• Repository: https://github.com/kardSIM/audio2img
• Demo: https://huggingface.co/spaces/youzarsif/audio2img

Results

<video controls src="https://cdn-uploads.huggingface.co/production/uploads/6674757dc1ccf20bff7089cf/6JL3EDXBcP-h26PX4T0zs.webm"></video> <img src="https://cdn-uploads.huggingface.co/production/uploads/6674757dc1ccf20bff7089cf/NmlyRozQileSdXUgpB_kW.png" width="400" height="300"> <video controls src="https://cdn-uploads.huggingface.co/production/uploads/6674757dc1ccf20bff7089cf/9ubrZlD1XD7xZrLgLAMh6.webm"></video> <img src="https://cdn-uploads.huggingface.co/production/uploads/6674757dc1ccf20bff7089cf/0QX906Q09xZYg5mJKFFwR.png" width="400" height="300">

<video controls src="https://cdn-uploads.huggingface.co/production/uploads/6674757dc1ccf20bff7089cf/QWP4ofLxT5zPf6p67Yh8H.webm"></video> <img src="https://cdn-uploads.huggingface.co/production/uploads/6674757dc1ccf20bff7089cf/8DJb-Cp650mIeb0VwJrmO.png" width="400" height="300">

Uses

The audio2img project aims to enhance generative modeling by integrating audio embeddings into the conditioning process of models like Stable Diffusion. This integration allows for the exploration of new creative possibilities by leveraging the rich semantic information contained in audio data.

Potential Users:

• Researchers and Developers

• Artists and Creatives

• Content Creators

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

https://huggingface.co/datasets/nateraw/fsd50k

Training Procedure

The core idea behind our training process is to achieve cross-modal alignment between audio and text embeddings using a two-stream architecture. This involves leveraging the powerful CLIPTextModel to generate text embeddings that serve as true labels for the audio embeddings produced by our Wav2Vec2Bert model.

Model Card Contact

• Email: youssefkardous1@gmail.com