sed-birdcode-ablation-ssl_beats_clip_pseudo

Sound Event Detection

Run using code available on Github

Pretrained sound event detection models focused on bioacoustics. Supports three main functions:

• Inference with pre-trained models: Within python, via a script, or via the large-scale inference (LSI) pipeline.
• Evaluation of model performance on detection datasets.
• Load pre-computed model detections for datasets like Xeno-Canto and iNaturalist.

Installation

Requires uv. Installation may take several minutes. GPU is not required but will improve speed.

Required packages are listed in pyproject.toml. To install them, run:

uv sync --group gpu   # omit --group gpu for CPU-only

All commands run through uv run. It may be necessary to include --group gpu if using a GPU. Evaluation and LSI also need the dataset storage referenced by configs/data/*.yml.

Large-scale inference and using precomputed selection tables both require alp-data, which is already included in pyproject.toml.

Quick start — BirdCODE over a folder of audio

Run the pretrained BirdCODE detector (loaded from the Hub) over every audio file in a folder — any sample rate, resampled to 32 kHz as needed — and write a selection table next to each recording: dir/x.wav → dir/BirdCODE_predictions/x.txt. Currently supports wav, flac, ogg, and mp3.

uv run sed-folder --folder /path/to/audio

Two short demo recordings are provided. To run BirdCODE on them, do:

uv run sed-folder --folder tests/samples/demo/audio

This writes tests/samples/demo/audio/BirdCODE_predictions/{20230730,20260623}.txt, which should match the tables in tests/samples/demo/output_expected/. On CPU it takes roughly 1.5 minutes after the model weights (~1.1 GB) are downloaded.

Postprocessing is applied: By default, per-frame detections are thresholded at 0.5, boxes with the same label are merged if separated by less than 1 second, and non-maximal suppression is applied with an IoU threshold of 0.8. Geography filtering is off by default; enable it with --geo-filter, a directory of *.gpkg range maps, and the recording site's coordinates (applied to every file):

uv run sed-folder --folder /path/to/audio \
    --geo-filter --range-map-dir geography/range_maps \
    --latitude 42.5 --longitude -72.2

Official models

CLI entry points

Every CLI has a describe subcommand that prints its config schema(s), e.g. uv run sed-eval describe.

Using BirdCODE in Python

FrameDetector loads a trained detector in-process, either from the HuggingFace Hub by repo id or from a checkpoint directory (local, gs://…, or r2://…):

from sound_event_detection.models import FrameDetector

# From the HuggingFace Hub (downloads the snapshot, then rebuilds the model);
birdcode = FrameDetector.from_hf_hub("EarthSpeciesProject/sed-birdcode").eval().to("cuda")

# Or from a checkpoint directory: weights from best_model.pt, labels from
# labels.txt, architecture from config.yaml.
ckpt = "checkpoints/birdcode_esp_research"
birdcode = FrameDetector.from_checkpoint_dir(ckpt, f"{ckpt}/config.yaml").to("cuda")

out = birdcode.run(audio, overlap=0.5)   # audio: np.ndarray [batch, samples] at 32 kHz
out.predictions                          # [batch, time, classes] probabilities in [0, 1]
out.class_names                          # list[str] labels aligned to the classes axis

Serving models

For large-scale inference and evaluation, we serve the model over HTTP, then point a client CLI at it via an http-client config.

A model config YAML tells the server what to load, dispatching on type. The unified server (sed-server) reads its path from the SED_MODEL_CONFIG environment variable.

Frame detectors — type: frame

Trained detectors (BirdCODE and ablations) loaded either from the HuggingFace Hub or from a local checkpoint directory. All current checkpoints run at 32 kHz.

Set hf_repo_id to download and serve a checkpoint from the Hub — this is how the example config loads BirdCODE. An optional revision pins a branch, tag, or commit (defaults to the repo's default branch):

type: frame
hf_repo_id: EarthSpeciesProject/sed-birdcode
# revision: main   # optional

Alternatively, model_folder serves a local checkpoint directory (expects config.yaml, best_model.pt, and labels.txt).

Serve either config the same way:

SED_MODEL_CONFIG=configs/birdcode/models/birdcode_esp_research.yml \
    uv run sed-server --host 0.0.0.0 --port 8100

sed-server accepts --host (default localhost), --port (default 8100), --workers, --reload, and --log-level. SED_DEVICE=cpu|cuda selects the device (default: cuda if available).

Ablation checkpoints use the same type: frame shape: configs/birdcode/models/ablations/.

Sliding-window detectors — type: perch2 | audioprotopnet | beats_sl_all

Clip classifiers wrapped in a SlidingWindowDetector to produce frame-level predictions. Each needs a backing classifier server already running, discovered through addr_file (a text file containing host:port):

type: audioprotopnet
addr_file: ~/audioprotopnet-server/server.addr
window_size: 5.0        # seconds
hop_size: 2.0           # seconds
analysis_window: 2.0    # optional; defaults to window_size

The external servers write their own server.addr; point the config's addr_file at it. Serve the wrapper the same way as a frame detector:

SED_MODEL_CONFIG=configs/birdcode/models/baselines/audioprotopnet_2s.yml \
    uv run sed-server --port 8100

beats_sl_all runs at 16 kHz — evaluate it with frame_eval_16k.yml (frame detection) or birdset_clip_eval_16k.yml (clip classification). Its backing classifier is served in-repo:

# 1. backing classifier (16 kHz), then record its host:port
SED_DEVICE=cuda uv run uvicorn \
    sound_event_detection.serving.sl_beats_all_server:app --host 0.0.0.0 --port 8200
echo "HOST:8200" > .server_addrs/beats_sl_all.addr   # path the config's addr_file points at

# 2. the sliding-window wrapper
SED_MODEL_CONFIG=configs/birdcode/models/baselines/beats_sl_all_2s.yml \
    uv run sed-server --port 8100

Denoising detector — type: denoising_detector

NOTE: This requires a separator server to be running. Separator server code will be provided at a later date.

Wraps a detector client and a source-separator client, adding POST /separate_and_detect (used by LSI) to the standard contract. Both backing servers must be up when it starts. Its model config names them as pure http-client configs:

type: denoising_detector
detector:  {url: http://localhost:8100, timeout: 300}   # a sed-server detector server
separator: {url: http://localhost:8200, timeout: 300}   # a separator server
threshold: 0.5
resampling_method: torchaudio_kaiser_fast

# with a detector server and a separator server already running:
SED_MODEL_CONFIG=configs/birdcode/models/denoising_detector.yml \
    uv run sed-denoising-server --host 0.0.0.0 --port 8110

sed-denoising-server takes the same options as sed-server (default port 8110).

HTTP contract

• GET / — model metadata: {labels, sample_rate, frame_rate, window_duration}
• GET /health — {status: "ok"} once the model is loaded
• GET /labels — ordered label list
• POST /run — frame-level inference; response {predictions, shape [batch, time, classes], frame_rate}
• POST /run_as_classifier — clip-level pooled inference; response shape [batch, classes]
• POST /separate_and_detect — denoising server only; per-stem audio + predictions

Evaluation — sed-eval

Serve a model, then run sed-eval against it with an eval config (what to evaluate) and an http-client config (how to reach the model — a url plus optional timeout/retries/auth; the client kind is auto-detected from the server).

# write an http-client config pointing at the running server, e.g.:
#   url: http://HOST:8100
uv run sed-eval --eval-config configs/birdcode/frame_eval.yml \
    --httpclient-config configs/birdcode/httpclient.yml \
    [--checkpoint-dir <dir>] [--output-dir <dir>]

• --checkpoint-dir — resumable checkpoint directory (auto-generated under checkpoints/sed/ if omitted).
• --output-dir — override the eval config's output_dir.
• sed-eval --resume <checkpoint-dir> — resume a run; configs are reloaded from the checkpoint.

Eval configs

An eval config selects the pathway through its dataset lists: frame_datasets (strong labels, with species_column) go through detection; clip_datasets (weak labels) through classification.

Metrics

• Frame pathway: frame mAP, event mAP per IoU threshold, thresholded precision/recall/F1.
• Clip pathway: cmAP (headline), cmAP5, mAP, pcmAP, MultilabelAUROC, top-1/top-3 accuracy, per-class AP, and gt_coverage.

Results

Each eval run writes <output_dir>/results.yaml, updated after every dataset:

• model — the served model's metadata (GET / response)
• frame_eval — the scoring parameters used
• frame_datasets.<name> — per-dataset detection metrics
• clip_datasets.<name> — per-dataset classification metrics

Large-scale inference (LSI)

Run a served detector over a dataset, persist per-recording results as compressed .npz shards, then postprocess (and optionally enrich) them into selection tables. Three stages: run → postprocess → features. Each stage takes --job-index N --num-jobs M to split the work across an array of parallel jobs, and writes a lineage.yaml chaining back to the stage that produced its input.

The LSI configs (configs/inference/lsi_birdcode_*.yml) run the BirdCODE frame detector over the full Xeno-Canto and iNaturalist training splits; they read their datasets from configs/data/inference/.

Run — sed-lsi

Builds a dataset from a run config (what to run) and a detector client from an http-client config (how to reach the model), then runs the sharded engine over this job's slice.

# with the appropriate server running (see below):
uv run sed-lsi --run-config configs/inference/lsi_birdcode_xc.yml \
    --httpclient-config <httpclient.yml> [--job-index N --num-jobs M] [--output-dir DIR]

The run config's output.detail selects what is stored per recording — and which server the url must reach:

Postprocess — sed-lsi-postprocess

Reads the combined predictions in each shard and writes a per-recording selection table (1:1 with the input shards). Re-postprocessing is a cheap re-run into a sibling directory.

uv run sed-lsi-postprocess --config configs/inference/lsi_birdcode_xc_postprocess.yml \
    --run-dir <run_dir> [--job-index N --num-jobs M]

--run-dir overrides the config's input.run_dir (postprocess several runs with one config).

Geography filtering

Setting postprocessing.geo_filter: true drops detections for species whose range maps exclude a recording's location (using the latitude/longitude stored in each shard). It requires postprocessing.range_map_dir — a directory (local path or cloud URI) of *.gpkg range-map files, globbed at startup and checked to exist before any shards are processed:

postprocessing:
  geo_filter: true
  range_map_dir: geography/range_maps   # dir of *.gpkg range maps

To use geography filtering, download the open range-map dataset from iNaturalist (https://www.inaturalist.org/pages/range_maps) into range_map_dir. Each range map's species name is resolved to a GBIF canonical name to match the detector's labels. The filter fails open: a detection is dropped only on positive out-of-range evidence (valid coordinates and a range map that excludes the point); recordings without coordinates, or species without a range map, are left untouched.

Features — sed-lsi-features

Enriches a postprocessed selection table with per-event v0minimal acoustic features. Writes enriched selection tables 1:1 with the postprocess shards.

uv run sed-lsi-features --config configs/inference/lsi_birdcode_xc_features.yml \
    --run-dir <run_dir> --postprocessing postprocessed_thr0.50_merge1.00_nms0.80_geo \
    [--job-index N --num-jobs M]

Loading a dataset with attached selection tables (Python)

Public GCS buckets hold BirdCODE detections as selection tables for a subset of Xeno-Canto and iNaturalist recordings. Two data configs load each corpus with those tables attached via the attach_lsi_selection_tables transform:

• configs/data/inference/xeno_canto_selection_tables.yml
• configs/data/inference/inaturalist_selection_tables.yml

Load either with alp_data.dataset_from_config, importing the transforms module first so the custom transform is registered:

import io
import pandas as pd
from alp_data import dataset_from_config
import sound_event_detection.data.transforms  # noqa: F401 — registers attach_lsi_selection_tables

dataset, meta = dataset_from_config("configs/data/inference/xeno_canto_selection_tables.yml")
print(meta["attach_lsi_selection_tables"])  # {'matched': ..., 'unmatched': ...}

# The attached `selection_table` column lives on the metadata backend
# (`dataset._data`), so you can read it without decoding audio. It is a TSV
# string (empty for unmatched rows); parse it into a DataFrame of events:
for row in dataset._data:
    if row["selection_table"]:
        events = pd.read_csv(io.StringIO(row["selection_table"]), sep="\t")
        break

Each row of a parsed selection_table is one detection event, with columns:

• Begin Time (s), End Time (s) — the event's span within the recording
• Species — predicted class label
• Score — mean BirdCODE probability over the event
• 13 v0minimal acoustic-feature columns (see sound_event_detection.inference.features_v0minimal.FEATURE_COLS)