esp-aves2-sed-birdcode-ablation-ssl-beats-clip-pseudo-encoder

Model Card for esp-aves2-sed-birdcode-ablation-ssl-beats-clip-pseudo-encoder

Encoder only. This repository contains the 32KHz BEATs backbone from the sed-birdcode-ablation-ssl_beats_clip_pseudo run. The classification head from that run is not included, and there is no label_map.json. See Relationship to the source checkpoint.

Quick start

from avex import load_model
backbone = load_model("esp_aves2_sed_birdcode_beats_encoder", device="cuda")

"esp_aves2_sed_birdcode_beats_encoder" is the official name (as per yaml file name based ontology)

Model Details

Model Description

esp-aves2-sed-birdcode-ablation-ssl-beats-clip-pseudo-encoder is an audio representation learning model (bioacoustic encoder) that produces frame-level embeddings for downstream bioacoustic tasks such as species classification and detection, individual identification, and vocal repertoire discovery.

It is the encoder from an ablation in a sound event detection (SED) line of work: a BEATs SSL backbone post-trained with clip-level pseudo-labels over a 7,475-taxon label space. Only the encoder is published here.

• Developed by: Earth Species Project
• Funded by: More info at https://www.earthspecies.org/about-us#support
• Shared by: Earth Species Project
• Model type: Audio representation learning model (Transformer; BEATs backbone)
• License: cc-by-nc-sa-4.0
• Finetuned from model: BEATs pretrained on AudioSet (see Parent Models)

Model Sources

• Repository: https://github.com/earthspecies/avex
• Paper: TBA
• Hugging Face Model: ESP-AVES2 Collection
• Configuration: train_config.yaml

Parent Models

1. BEATs (pretrained on AudioSet)
   • Source: https://github.com/microsoft/unilm/tree/master/beats
   • Description: Self-supervised transformer audio encoder used as the base SSL checkpoint.
   • License: See upstream repository

Input Contract (important)

This model does not use the AVEX defaults. Two settings are easy to conflate and both matter:

sample_frequency is not a declaration of the incoming audio's rate. It fixes the analysis window (25 ms x 16000 = 400 samples), the hop (160 samples), and the mel filterbank (257 FFT bins x 128 mels). All three are frozen in this checkpoint as the buffers backbone.fbank.window (400,) and backbone.fbank.mel_fb (257, 128). Setting it to 32000 rebuilds them at (800,) and (513, 128), and the checkpoint then fails to load with a size mismatch on both.

Consequence of 32 kHz audio through a 16 kHz fbank: each window spans 12.5 ms of real time, the hop is 5 ms, and content at true frequency f is analysed at f/2. So 5.0 s yields 998 frames (the count a 10 s clip gives at 16 kHz), and bird song at 2-8 kHz lands at 1-4 kHz. The encoder was trained on exactly these features.

Two distinct failure modes to avoid:

• Resampling input to 16 kHz — loads fine, silently degrades the representation.
• Changing sample_frequency to 32000 — will not load at all.

5.0 s at 32 kHz produces 496 tokens (62 time patches x 8 frequency patches) of dimension 768.

Uses

Direct Use

Feature extraction for bioacoustic tasks: species classification and detection, repertoire and individual classification, retrieval and clustering.

Downstream Use

Linear probing, retrieval, and clustering; can also be fine-tuned for task- and domain-specific bioacoustic applications.

Out-of-Scope Use

This is an encoder with no classification head. It produces no class predictions and no text output. Using it as a stand-alone classifier without training a probe is out of scope.

Bias, Risks, and Limitations

• Bias: Training data relies heavily on citizen-science recordings and may over-represent certain taxa and regions (e.g. Northern Hemisphere), which can impact generalization.
• Risks: Embeddings can be misused for harmful wildlife exploitation (e.g. locating endangered species) if deployed without safeguards.
• Limitations: The fixed 32 kHz / 5.0 s input contract above is a hard constraint. Performance can degrade under large distribution shifts (habitat, device, background noise). As an ablation checkpoint, it has not been benchmarked to release standard — see Evaluation.

Recommendations

Use as an encoder (feature extractor) and validate performance on your target domain. For sensitive deployments, apply access controls and follow conservation best practices.

How to Get Started with the Model

Loading this model requires the AVEX (Animal Vocalization Encoder) library avex to be installed.

Installation

pip install avex

Or with uv:

uv add avex

Loading the Model

This repository has no packaged config name in avex, so load it by checkpoint_path. AVEX resolves hf:// URIs directly:

import torch
from avex import load_model
from avex.configs import AudioConfig, ModelSpec

REPO = "EarthSpeciesProject/esp-aves2-sed-birdcode-ablation-ssl-beats-clip-pseudo-encoder"
CKPT = f"hf://{REPO}/esp-aves2-sed-birdcode-ablation-ssl-beats-clip-pseudo-encoder.safetensors"

spec = ModelSpec(
    name="beats",
    pretrained=False,
    device="cuda",
    # See train_config.yaml for the full init_config; sample_frequency stays 16000.
    audio_config=AudioConfig(
        sample_rate=32000,
        representation="raw",
        normalize=False,
        target_length_seconds=5,
    ),
)

backbone = load_model(spec, device="cuda", checkpoint_path=CKPT, return_features_only=True)

Expect 252/252 params matched, 0 unexpected in the load log. load_model detects that the checkpoint has no classifier and selects embedding mode automatically, so return_features_only=True is optional.

Using the Model

# audio_tensor: (batch, 160000) float waveform at 32 kHz  (5.0 s)
with torch.no_grad():
    tokens = backbone(audio_tensor)
    # Shape: (batch, 496, 768) -- 62 time patches x 8 frequency patches

# Fixed-size embedding
embedding = tokens.mean(dim=1)   # (batch, 768)

Tokens are flattened time-major: index t * 8 + f. To keep the frequency structure, regroup before pooling:

b, n, d = tokens.shape
freq_concat = tokens.reshape(b, n // 8, 8, d).mean(1).reshape(b, -1)  # (batch, 6144)

This (batch, 6144) readout is the representation the source run's discarded head consumed, so it is a good starting point for downstream probes. Note it is not one of AVEX's built-in aggregation modes ("mean" / "max" / "cls_token" / "none" all collapse time and frequency together into 768).

Transfer Learning with Probes

from avex.configs import ProbeConfig
from avex.models.probes import build_probe_from_config

probe_config = ProbeConfig(
    probe_type="linear",
    target_layers=["last_layer"],
    aggregation="mean",
    freeze_backbone=True,
    online_training=True,
)

probe = build_probe_from_config(
    probe_config=probe_config,
    base_model=backbone,
    num_classes=10,   # your number of classes
    device="cuda",
)

Class Label Mapping

Not applicable. This is an encoder-only release with no classification head, so no label_map.json is provided.

Relationship to the source checkpoint

The source run sed-birdcode-ablation-ssl_beats_clip_pseudo stored its backbone under an encoder._model.backbone.* prefix and carried a Linear(6144, 7475) head as top-level classifier.{weight,bias}. Publishing this encoder involved exactly two changes, both lossless for the backbone:

1. Key prefix encoder._model.backbone.* -> backbone.*, matching avex.models.beats_model.Model.
2. The head was dropped (252 backbone tensors kept, 2 head tensors removed).

Backbone weights are bitwise identical to the source checkpoint; verified by a strict=True load and by comparing forward-pass outputs.

The dropped head was a frame-level SED classifier over the 8 concatenated frequency patches (6144 = 8 x 768), applied per time patch and pooled over time. It is not published here and is not required to use these embeddings.

Training Details

Training Data

TBA — the pseudo-label source and dataset composition for this ablation are not recorded in the artifacts used to build this repository. The label space was 7,475 taxa spanning birds, insects, and amphibians.

Training Data Sources

TBA

Training Procedure

• Stage 1 (SSL): BEATs pretrained on AudioSet.
• Stage 2: post-training with clip-level pseudo-labels over a 7,475-taxon label space (clip_pseudo), as one arm of a SED ablation study.
• Augmentations: TBA

Training Hyperparameters

TBA — see train_config.yaml for the inference-time model configuration. That file is a model spec for loading, not a full training config, so it does not record optimizer or schedule settings.

Evaluation

TBA — this is an ablation checkpoint and has not been evaluated on the ESP-AVES2 benchmark suite. Do not assume parity with released ESP-AVES2 models.

For reference, the sibling released model esp-aves2-sl-beats-all carries full BEANS / BirdSet / Individual ID / Vocal Repertoire results.

Testing Data, Factors & Metrics

TBA

Results

TBA

Environmental Impact

TBA

Technical Specifications

Model Architecture and Objective

BEATs transformer encoder, 12 layers, 768-dimensional, deep_norm=True, no label predictor. 252 weight tensors.

Key components:

• Encoder: BEATs transformer (12 layers, 768 dim, 12 heads, FFN 3072)
• Feature extraction: internal batched fbank, 128 mel bins, 16 kHz geometry (see Input Contract)
• Patch embedding: 16 x 16, stride 16 -> 8 frequency patches
• Output: (batch, 496, 768) for a 5.0 s clip at 32 kHz

Compute Infrastructure

TBA

Model Configuration

Model configuration is available in train_config.yaml.

Citation

TBA — no publication is associated with this ablation checkpoint. For the ESP-AVES2 model family, cite:

@inproceedings{miron2025matters,
  title={What Matters for Bioacoustic Encoding},
  author={Miron, Marius and Robinson, David and Alizadeh, Milad and Gilsenan-McMahon, Ellen and Narula, Gagan and Chemla, Emmanuel and Cusimano, Maddie and Effenberger, Felix and Hagiwara, Masato and Hoffman, Benjamin and Keen, Sara and Kim, Diane and Lawton, Jane K. and Liu, Jen-Yu and Raskin, Aza and Pietquin, Olivier and Geist, Matthieu},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026}
}

Glossary

• Bioacoustic encoder: A model that maps audio to embeddings useful for downstream bioacoustic tasks.
• Linear probing: Training a simple linear model on frozen embeddings to assess representation quality.
• SED: Sound Event Detection; prediction of event presence over time within a clip.
• Clip pseudo-label: A clip-level label produced by another model rather than by human annotation.

More Information

• Project page: TBA
• Documentation: TBA
• Issue tracker: https://github.com/earthspecies/avex/issues

Model Card Authors

• Earth Species Project

Model Card Contact

Contact: [email protected]