harmonia-multimodal-music-intelligence

Harmonia — Multimodal Music Intelligence Stack

Author: Moyosore Ogunjobi
What this is: A masters-level, end-to-end system for music tagging, emotion estimation, text-to-audio search, and recommendation, trained from multiple datasets and stitched together with a calibrated Mixture-of-Experts fusion model.

Why Harmonia exists

Music ML is usually siloed: tagging models ignore symbolic structure, recommenders ignore audio, and search systems ignore user behavior. Harmonia unifies these signals into a shared representation and exposes them through a single pipeline.

What it can do

• Auto-tagging: genre / instrument / mood-theme (multi-label)
• Emotion regression: valence + arousal (continuous)
• Text ↔ audio retrieval: search audio by natural-language prompts
• Recommendation: top-K ranking with LightGCN trained on playcount graphs
• Fusion MoE: combines specialized experts into a calibrated final predictor

System overview (6 trainable models)

1. AudioTagger — CLAP/HTSAT audio encoder + multi-label head
2. AudioEmotion — shared audio encoder + valence/arousal head
3. ContrastiveAdapter — InfoNCE alignment (audio ↔ tag-captions)
4. SymbolicMusicBERT — MIDI-based understanding backbone + task head
5. RecLightGCN — collaborative filtering on user-song graphs
6. FusionMoE — mixture-of-experts + temperature calibration

Datasets

Place datasets under data/raw/:

• MTG-Jamendo: data/raw/mtg_jamendo/
• FMA: data/raw/fma/
• DEAM: data/raw/deam/
• Lakh MIDI: data/raw/lakh_midi/
• MSD Taste Profile: data/raw/msd_taste_profile/

Run:

python scripts/check_data.py

Install

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Train (each model separately)

python -m harmonia.training.train_audio_tagger --config configs/audio_tagger_small.yaml
python -m harmonia.training.train_audio_emotion --config configs/audio_emotion_small.yaml
python -m harmonia.training.train_contrastive_adapter --config configs/contrastive_small.yaml
python -m harmonia.training.train_musicbert --config configs/musicbert_small.yaml
python -m harmonia.training.train_lightgcn --config configs/lightgcn_small.yaml
python -m harmonia.training.train_fusion_moe --config configs/fusion_small.yaml

Or run everything:

bash scripts/train_all.sh

Build the retrieval index (FAISS)

bash scripts/build_faiss.sh

Demo (Gradio)

python app.py

Results

This section is populated by saved metrics files from the training scripts:

• Tagging: mAP / ROC-AUC
• Emotion: RMSE / CCC
• Retrieval: Recall@10 / MRR
• Rec: Recall@20 / NDCG@20

How to use the pipeline (Python)

from harmonia.inference.pipeline import HarmoniaPipeline

pipe = HarmoniaPipeline.from_pretrained("./outputs/harmonia_bundle")
out = pipe.predict(audio_path="example.wav", text_query="sad jazzy piano at night")
print(out["tags"], out["emotion"], out["retrieval_topk"][:5])

License & data policy

This repo ships code + trained weights only. Datasets remain under their original licenses. Do not redistribute copyrighted audio, MIDI, or lyrics.