Localized time-frequency representation learning for bioacoustic classification in complex soundscapes

arXiv:2502.13440 · cs.SD, cs.AI, cs.CV, eess.AS, q-bio.QM · Submitted 2025-02-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Localized time-frequency representation learning for bioacoustic classification in complex soundscapes".

Jane: A semi-supervised acoustic bird detector is proposed that enables the detection of time-overlapping vocalizations without requiring extensive expert-labeled datasets, making it valuable for monitoring rare species in complex soundscapes.

Tom: First, who's behind it and why it matters.

Paper summary: Lu: To wrap up our discussion on "Localized time-frequency representation learning for bioacoustic classification in complex soundscapes," the authors present a methodology that systematically addresses the challenges of time-overlapping vocalizations and limited labeled data through a four-step process involving segmentation, compression, embedding, and classification. This approach demonstrates how structured AI techniques can handle messy acoustic reality effectively without relying on prohibitively large datasets for every single specific task.

Meng: In essence, the paper shows that by focusing on creating robust embeddings that are invariant to time shifts and using self-supervised contrastive learning derived from natural sound propagation variability, we can build a detection system capable of handling the complexity of real-world soundscapes quite well.

Lalam: The most impactful vision I get from this is how this kind of AI advancement could improve our understanding and management of natural cultures, as it enables continuous monitoring that shifts the focus from laborious data gathering to high-level analysis and conservation strategy.

Tom: So, the paper's title itself really summarizes the core contribution: localizing time-frequency representation learning for bioacoustic classification in complex soundscapes, which speaks directly to its practical application in messy field conditions.

Jane: It’s important to remember that while they report an F0 point 5 score of zero point seven zero one across three hundred fifteen classes from one hundred ten species on a hold-out set, this performance is achieved by combining multiple sophisticated techniques, including the bird-pass filter and sink class to enhance robustness against false positives in continuous field recordings.

Lu: The implication for future work seems to be exploring how this localized representation learning can be adapted for even more diverse acoustic environments or perhaps integrating environmental data more deeply into the embedding process to further refine the classification.

Meng: From a practical standpoint, the next step is likely moving from this successful proof-of-concept on a specific site like Singapore to developing deployable hardware that needs to run efficiently in remote areas with minimal maintenance.

Lalam: Ultimately, this work demonstrates that smarter AI techniques are becoming capable of handling complex bioacoustic tasks using less direct human intervention, which has deep implications for how we approach ecological data collection and management moving forward.

Conclusion: Tom: So we've been diving deep into how this paper tackles those tricky overlapping bird calls using segmentation and contrastive learning to create these robust embeddings, and now we get to look at the final thoughts on "Localized time-frequency representation learning for bioacoustic classification in complex soundscapes."

Jane: That title really tells the story of where this research is focused—it’s all about taking those specific sound patterns from a place like Singapore and making them understandable by an AI, which is fascinating.

Lu: I think the core idea is that we don't need to label every single sound to train a system; instead, we can learn how sounds are structured locally in time and frequency first.

Meng: From my end, the implication for practical use seems pretty straightforward: this could mean monitoring rare species in really noisy environments without needing a team of experts just to collect all that initial data.

Lalam: I see it as a way for us to build a more intuitive understanding of the acoustic richness of our planet, potentially revealing hidden patterns in biodiversity that we haven't even considered yet.

Tom: Exactly! It’s about making complex soundscapes manageable for AI analysis, which is huge because field data is messy and overwhelming.

Jane: And the authors really managed to put together a method that handles those time overlaps by splitting calls into segments first, which simplifies the problem immensely for the subsequent learning stages.

Lu: The way they use self-supervised contrastive learning to teach the model what makes two sounds similar, even if they happen at slightly different times, is a very clever way to build that invariant representation.

Meng: I'm interested in how this localized approach scales; if we can adapt this to different acoustic conditions, it opens up possibilities for deploying monitoring systems in much more challenging locations.

Lalam: Thinking about the bigger picture, this kind of AI advance could fundamentally improve our ability to track and protect wildlife across diverse habitats globally.

Tom: So, we've seen the mechanics and the results; what does this actually mean for conservation efforts on a grand scale? What’s the real-world impact here?

National University of Singapore · Department of Electrical & Computer Engineering · National Parks Board, Singapore

cs.SD, cs.AI, cs.CV, eess.AS, q-bio.QM

Submitted: 2025-02-19

Updated: 2026-09-28

Code: https://github.com/kahst/BirdNET-Analyzer

Importance score: 72/100

The gist: A semi-supervised acoustic bird detector is proposed that enables the detection of time-overlapping vocalizations without requiring extensive expert-labeled datasets, making it valuable for

Key concepts

Time-Frequency Representation (TFR): TFRs are a matrix representation of an audio signal that shows how energy is distributed across different frequencies over time. The method extracts these TFRs from raw recordings, focusing on high-energy regions while filtering out noise and non-bird sounds.
Contrastive Learning: This self-supervised technique learns representations by comparing pairs of similar sounds, such as those recorded on the two microphones of a single unit. It forces the model to create embeddings where similar sounds are close together in the embedding space, improving classification performance without needing extensive labels.
Semi-supervised Learning: This approach trains a final classifier using only a small set of labeled data after an initial self-supervised training phase on massive amounts of unlabeled data. The model learns general features from the large soundscape recordings first, making it effective for detecting rare species with minimal expert labeling.
Embedding Space: This is the high-dimensional mathematical space where the compressed TFRs are mapped. The contrastive learning objective ensures that sounds with similar acoustic characteristics are positioned close together in this space, allowing the final classifier to easily distinguish between different bird species.

Terminology

Contrastive Learning — 12× in this paper · explained in 5 episodes
Loss Function — 8× in this paper · explained in 2 episodes
Accuracy — 7× in this paper · explained in 1 episode
Representation Learning — 6× in this paper · explained in 2 episodes
Clustering — 5× in this paper · explained in 2 episodes
Embeddings — 5× in this paper · explained in 2 episodes
Latent Representation — 4× in this paper · explained in 2 episodes
Adam — 3× in this paper · explained in 2 episodes
Latent Space — 3× in this paper · explained in 7 episodes
Robustness — 3× in this paper · explained in 5 episodes
Transfer Learning — 3× in this paper · explained in 10 episodes
Verification — 3× in this paper · explained in 2 episodes
Analyzer — 2× in this paper · explained in 1 episode
Calibration — 2× in this paper · explained in 7 episodes
Core — 2× in this paper · explained in 2 episodes
Few-Shot Learning — 2× in this paper · explained in 4 episodes
Self-Supervised Learning — 2× in this paper · explained in 8 episodes
Activation Function — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Data Augmentation — 1× in this paper · explained in 7 episodes
Feature Extraction — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Initial Data — 1× in this paper · explained in 2 episodes
Meta-learning — 1× in this paper · explained in 2 episodes
Ontology — 1× in this paper · explained in 2 episodes
Selection Bias — 1× in this paper · explained in 4 episodes
Singapore Botanic Gardens — 1× in this paper

Summary

A semi-supervised acoustic bird detector is proposed that enables the detection of time-overlapping vocalizations without requiring extensive expert-labeled datasets, making it valuable for monitoring rare species in complex soundscapes.

How it works

The proposed method consists of four main steps: 1) Segmentation, 2) Data compression, 3) Embedding, and 4) Classification. Step 1 involves extracting individual bird calls using an energy-based segmentation technique. This isolation limits noise and allows time-overlapping calls to be treated separately as long as they do not overlap in frequency. A consequence of this approach is that single calls/songs may be split into multiple segments.

Step 2 is self-supervised, where the network learns a compressed representation of the segments while retaining most of the information. Step 3 uses this representation to learn a new embedding, ensuring both translational invariance and that similar sounds have similar embedding – two key properties for efficient clustering and classification.

Data Collection and Representation

The data collection involved recordings from two different locations in the Singapore Botanic Gardens (SBG) over seven months, utilizing three Wildlife Acoustics SongMeter 4 TS recorders deployed at both sites. The recording setup at site number one allowed the same call to be detected on multiple recorders, which was leveraged later for training the contrastive network.

The time-frequency representation (TFR) is extracted through a multi-step process:

Compute a spectrogram with 2,048 FFT bins, a Hamming window, and an overlap of 1,536 samples between windows. Retain frequency bins between 500 Hz and 15 kHz only.
Convert the spectrogram to dB using the inter-quartile range for noise variance estimation to adaptively extract regions with significantly higher energy than the background noise at that frequency.
Reduce frequency resolution by a factor of 5 by max-pooling.
Blank out time bins with low variance across broad frequency bands, as these represent impulsive sounds not characteristic of birds.
Perform a watershed segmentation to obtain disconnected regions of high energy in the spectrogram.

The resulting TFRs are then converted to a constant duration of 2.7 seconds (256 time bins) for algorithms requiring fixed input size through random selection or zeropadding, ensuring the complexity is not impacted by call density as long as single TFRs do not capture multiple calls.

Self-Supervised Learning and Contrastive Representation

The auto-encoding stage learns a compressed representation of the TFR using a convolutional deep auto-encoder, constrained to a latent representation of only 512 values. This learning is self-supervised, requiring no labeled data is required. The encoder section is then kept as a pre-trained processor.

Next, contrastive learning discovers an embedding invariant to time translation and where similar sounds have similar representations. Instead of random augmentation, sample pairs are derived from recordings of the same bird vocalization on the two microphones connected to each recorder unit, using natural sound propagation variability as the desired augmentation. The loss function is designed such that it maximizes similarity within each pair while inducing orthogonality for non-paired representations, supporting up to 1024 distinct classes.

Supervised Refinement and Classification

The final stage involves a classifier with four dense layers and batch normalization, taking the embedding vector as input to assign a confidence score to each predefined class. This classifier is trained on curated labeled data, initially extracted from Xeno-Canto files (about 2/3 of the training data).

To enhance robustness against false positives in continuous field recordings, two additions are made:

A separate bird-pass filter is trained to distinguish bird sounds from generic non-bird sounds. This filter is trained on TFRs extracted from the auto-encoder and uses TFRs from the UrbanSound8K dataset and marine mammal vocalizations for the non-bird class.
An additional sink class is added, trained on local sounds from the SBG, to help the model learn lower-level features for classes easily confused with other local sounds.

The model is trained end-to-end without freezing any preceding layers, simultaneously training with both the primary cross-entropy loss and an auxiliary contrastive loss to reinforce the need for paired samples to have high similarity.

Results and Evaluation

On a hold-out test set of 315 bird classes across 110 species, the model achieved a mean F0.5 score of 0.701, outperforming BirdNET on this test set despite significantly fewer labeled training samples. The performance for individual bird classes was skewed, with many achieving F0.5 scores above 0.9 while others scored 0.0, indicating confusion with similar classes or low similarity between test and training samples.

Improvements for AI systems

Here are specific improvements to existing AI systems derived from the proposed semi-supervised acoustic bird detector, along with what those improved systems could achieve:

Improvement: Integration of Self-Supervised Contrastive Learning for Few-Shot Species Discovery and Novel Class Detection.
Improvement: Implementation of a robust Bird-Pass binary classifier trained on urban/anthropogenic soundscapes (UrbanSound8K, marine mammal vocalizations) to handle false positives in continuous monitoring data.
Improvement: Enhancement of the Feature Representation via Auto-encoding and Contrastive Embedding Space to capture fine temporal features and enhance translation invariance for robust classification.
Improvement: Utilization of an End-to-End Training Strategy where the embedding network is fine-tuned simultaneously with the primary classifier (using alternating cross-entropy loss and contrastive loss) to ensure representations remain invariant while optimizing for class separation.
The improved system can perform high-accuracy bird species classification using only a few labeled training samples per class (e.g., 11 labeled samples per class, as reported), significantly reducing the need for extensive manual annotation efforts compared to state-of-the-art models like BirdNET.
The system can effectively detect and classify time-overlapping bird vocalizations (separated in frequency) within complex, noisy soundscapes by leveraging energy-based segmentation and contrastive learning on microphone pairs, allowing continuous monitoring without requiring perfectly isolated signals.
The improved system can operate reliably in highly dynamic urban or ecologically rich environments (like Singapore's SBG) by maintaining high precision (>0.80 for many classes) even when exposed to a massive variety of sounds, including anthropogenic noises and non-target fauna, through the learned sink class mechanism and the dedicated bird-pass filter.
The system can be used for rapid biodiversity assessment in remote or newly surveyed regions where labeled data is scarce, enabling the automated discovery and annotation of novel vocalization classes by clustering latent embedding vectors from raw recordings.

Sources

A Simple Framework for Contrastive Learning of Visual Representations