live-vs-studio-classifier

Live-vs-studio music classifier

A neural network that listens to a piece of music and predicts whether the recording is a live recording or a studio recording.

• 🎙️ Try the browser demo — runs entirely client-side via transformers.js, no server, no uploads.
• 💻 GitHub repository — source code, training data sources, deployment notes.
• 📊 Detailed comparison vs. five zero-shot audio LLMs — methodology, prompt iteration, failure-mode analysis.

Headline numbers

The 0.5-pp drop on the adversarial test set is small — the model generalizes well. It doesn't rely on crowd-noise shortcuts: soundboard-mixed live recordings (no audible crowd) still score 93.1%.

Quick start

Python (recommended)

# Install
pip install librosa torch transformers huggingface_hub

# Download the model files and predict
from huggingface_hub import snapshot_download
import sys

repo = snapshot_download("kaaaaan/live-vs-studio-classifier")
sys.path.insert(0, repo)
import predict

classifier = predict.Classifier()
result = classifier.classify("path/to/song.mp3")
print(f"{result['label']} ({result['confidence']:.2f})")
# -> live (0.91)

Or as a one-liner CLI:

python -c "
from huggingface_hub import snapshot_download
import subprocess, sys
repo = snapshot_download('kaaaaan/live-vs-studio-classifier')
subprocess.run([sys.executable, f'{repo}/predict.py', 'song.mp3'])
"

Example output:

song.mp3
  prediction: live
  confidence: 0.913  (3/3 live)
  per-window p(live): 0.872 0.940 0.928
  (three windows at 25/50/75% of usable span (240.0s clip))

The first call downloads the AST backbone (~350 MB from MIT/ast-finetuned-audioset-10-10-0.4593) and caches it; subsequent calls are fast. Apple Silicon and CUDA are auto-detected; pass device="cpu" to force CPU.

Browser (transformers.js / ONNX Runtime Web)

The repository ships an int8-quantized ONNX backbone (ast_backbone_int8.onnx, 89.5 MB) and JSON-encoded probe weights (probe-weights.json) for browser-side inference. The live demo at https://ziipo.github.io/LiveAudioClassifier/ uses exactly these files. See web/ in the GitHub repo for the reference implementation.

How it works

The classifier is a two-part model:

1. AST backbone (MIT/ast-finetuned-audioset-10-10-0.4593) — 86M parameters, pretrained on AudioSet to understand audio at a general level. Frozen during training.
2. Linear probe — 1,538 parameters. A single nn.Linear(768, 2) layer trained on the AST embeddings to make the live-vs-studio decision.

The probe weights are in probe-weights.json (~42 KB). The AST backbone is shipped as an int8-quantized ONNX file (~89 MB) for browser use; Python users load the original PyTorch backbone via transformers.

For each input clip, the pipeline takes three 30-second windows at 25/50/75% of the usable span, runs each through AST + the probe, and takes the majority vote.

What it was trained on

24,227 30-second audio clips, split ~80/10/10 into train/val/test at the track level (no clip leakage between splits):

Training script in the GitHub repo; not bundled here. Total training time was a few seconds on Apple Silicon MPS — the AST embeddings were extracted once and cached, and only the 1,538-parameter probe was trained.

How it compares to other approaches

Head-to-head against five recent open-weights audio LLMs and one alternative encoder on the same stratified 500-clip subset (250 live / 250 studio, seed 42), one 30-second clip at a time:

MERT-pretraining is competitive with but not clearly better than AudioSet-pretraining for this task. The MERT probe wins 1.6 pp on the matched subset, ties on the full test split, and costs ~9× more inference latency and ~3.7× more parameters than the AST setup, with no quantized browser-ready export path, so AST is what's included here.

A note on the numbers: the headline 94.46% and 93.93% accuracies above are per-song majority-voted across three windows per song (how the model is actually used). The comparison table here is per-clip — one 30-second window scored independently — because that's how the LLMs were tested. The AST's per-clip number on the full 2,401-clip test split is 95.5%; on the matched 500-clip comparison subset it's 94.4%.

Limitations

• Studio recordings made to sound live (reverb-heavy production, room mics, single-take sessions) can be misclassified as live. Very clean soundboard-style live recordings can occasionally go the other way.
• Short audio under 5 seconds is rejected; 5-30s is padded to 30s and gets a single window; 60s+ gets the full three-window treatment.
• First call is slow while the AST backbone downloads / loads (~350 MB PyTorch / ~89 MB ONNX). Cached afterward.

License

MIT for the probe weights, inference code, and configuration. The AST backbone (MIT/ast-finetuned-audioset-10-10-0.4593) is also MIT. Training data sourced from external sources (FMA: CC-licensed per-track; Internet Archive Live Music Archive: per-band CC variants) is not redistributed in this repository.

Acknowledgements

• MIT CSAIL for the Audio Spectrogram Transformer
• Xenova / HuggingFace for the pre-converted ONNX variants used in the browser path
• transformers.js for browser inference machinery
• The Internet Archive Live Music Archive for ~10,000 live clips
• The Free Music Archive for the studio counterpart

Citation

@misc{liveaudioclassifier-2026,
  title  = {Live-vs-studio music classifier (AST + linear probe)},
  author = {Ken},
  year   = {2026},
  url    = {https://huggingface.co/kaaaaan/live-vs-studio-classifier},
}