clearspeech-unet

🎙️ ClearSpeech

AI-Powered Speech Enhancement & Transcription System

ClearSpeech uses a custom U-Net deep learning model to remove background noise from audio, then transcribes the enhanced audio using OpenAI's Whisper. Perfect for cleaning up voice recordings, meeting audio, podcasts, or any noisy speech.

🌐 Live Website (will be updated): https://clearspeech.yourdomain.com

🌟 Features

• 🧹 AI-Powered Noise Reduction: Custom U-Net model trained to remove background noise
• 📝 Automatic Transcription: Whisper integration for accurate speech-to-text
• ⚡ Fast Processing: Optimized pipeline with GPU support
• 🌐 REST API: Easy-to-use FastAPI backend
• 🎯 High Quality: Val loss of 0.031
• 🔧 Flexible: Enhancement-only, transcription-only, or both

📋 Table of Contents

• Installation
• Quick Start
• API Documentation
• Project Structure
• Contributing

🚀 Installation

Prerequisites

• Python 3.8+

• pip

• Optional CUDA GPU

Step 1: Clone Repository

git clone https://github.com/yourusername/ClearSpeech.git
cd ClearSpeech

Step 2: Create Virtual Environment

# Create environment 
python3.10 -m venv venv 
# Activate (macOS/Linux) 
source venv/bin/activate 
# Activate (Windows) 
venv\Scripts\activate

Step 3: Install Dependencies

# Install dependencies
pip install -r requirements.txt

Step 4: Download Pretrained Model

# Download model
python -c "
from huggingface_hub import hf_hub_download
hf_hub_download(
    repo_id='thecodeworm/clearspeech-unet',
    filename='best_model.pt',
    local_dir='enhancement_model/checkpoints/'
)
"

Step 5: Generate Noisy Samples

1. Make your own WAV sample
2. Run the generate_noisy_samples.py file on the sample to make the audio noisier to test the model

# Generate all noise types at multiple SNR levvels
python generate_noisy_samples.py \
 --input my_clean_voice.wav \
 --output test_samples/

⚡ Quick Start

Method 1: Using the API (Recommended)

Start the server:

python -m backend.app

Server starts at http://localhost:8000

Start the server:

cd frontend
python -m http.server 3000

Frontend starts at http://localhost:3000

Process audio:

# Full pipeline (enhance + transcribe)
curl -X POST "http://localhost:8000/process" \
  -F "file=@your_audio.wav" \
  | jq .

# Enhance only
curl -X POST "http://localhost:8000/enhance" \
  -F "file=@your_audio.wav" \
  -o enhanced_output.wav

# Transcribe only
curl -X POST "http://localhost:8000/transcribe" \
  -F "file=@your_audio.wav" \
  -F "enhance=true" \
  | jq .

Method 2: Using Python

from backend.inference_pipeline import EnhancementPipeline

# Initialize pipeline
pipeline = EnhancementPipeline(
    cnn_checkpoint_path="enhancement_model/checkpoints/best_model.pt",
    whisper_model_name="base",
    device="cpu"  # or "cuda" or "mps"
)

# Process audio
result = pipeline.process("path/to/noisy_audio.wav")

print(f"Transcript: {result['transcript']}")
print(f"Duration: {result['duration']:.2f}s")

# Save enhanced audio
import soundfile as sf
sf.write("enhanced.wav", result['enhanced_audio'], result['sample_rate'])

Method 3: Command Line

# Enhance audio file
python enhancement_model/infer.py \
  --checkpoint enhancement_model/checkpoints/best_model.pt \
  --input noisy_audio.wav \
  --output enhanced_audio.wav \
  --comparison  # Creates stereo comparison file

📚 API Documentation

Interactive Docs

Once the server is running, visit:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Endpoints

POST /process

Process audio with enhancement and transcription.

Request:

  -F "[email protected]" \
  -F "language=en" \
  -F "skip_enhancement=false"

Response:

{
"success": true,
"transcript": "Transcribed text here",
"duration": 3.5,
"language": "en",
"enhanced_audio_url": "/download/enhanced_123.wav",
"segments": [...],
"processing_time": 2.3
}

POST /enhance

Enhance audio only (no transcription).

Request:

curl -X POST "http://localhost:8000/enhance" \
  -F "[email protected]" \
  -o enhanced.wav

Response: Enhanced audio file (WAV)

POST /transcribe

Transcribe audio with optional enhancement.

Request:

curl -X POST "http://localhost:8000/transcribe" \
  -F "[email protected]" \
  -F "language=en" \
  -F "enhance=true"

Response:

{
"success": true,
"transcript": "Transcribed text",
"duration": 3.5,
"language": "en",
"segments": [...]
}

GET /download/{filename}

Download enhanced audio file.

GET /health

Health check endpoint.

📁 Project Structure

ClearSpeech/
├── backend/                    # FastAPI backend
│   ├── app.py                 # Main API server
│   ├── inference_pipeline.py  # Processing pipeline
│   └── requirements.txt
├── enhancement_model/          # U-Net model
│   ├── model.py               # U-Net architecture
│   ├── dataset.py             # PyTorch dataset
│   ├── train.py               # Training script
│   ├── infer.py               # Inference script
│   ├── checkpoints/           # Trained models
│   │   └── best_model.pt
│   └── requirements.txt
├── data/                       # Training/test data
│   ├── audio_clean/           # Clean audio
│   ├── audio_raw/             # Noisy audio
│   ├── metadata/
│   │   └── metadata.json      # Dataset metadata
│   └── spectrograms/          # Mel-spectrograms
│       ├── clean/
│       └── noisy/
├── frontend/                   # Web interface (optional)
│   ├── index.html
│   └── script.js
├── tests/                      # Test files
│   └── test_backend.py
├── README.md
└── requirements.txt

🤝 Contributing

We welcome contributions! Here's how:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open a Pull Request

Development Setup

# Install dev dependencies
pip install -r requirements-dev.txt

# Run tests before committing
python -m pytest tests/

# Format code
black backend/ enhancement_model/

🙏 Acknowledgments

• U-Net Architecture: Inspired by Ronneberger et al.
• Whisper: OpenAI Whisper
• Training Data: LibriSpeech, MS-SNSD

📧 Contact

Project Maintainers: Aditya Chanda, Josh Pal, Advik Kumar Singh

Project Link: https://github.com/thecodeworm/ClearSpeech

⭐ Show Your Support

Give a ⭐️ if this project helped you!

---

Built with ❤️ using PyTorch and FastAPI