Time-frequency localization of bird calls in dense soundscapes

Passive acoustic monitoring enables large-scale observation of wildlife, but most bioacoustic classifiers only predict species presence in a time window without localizing vocalizations precisely in time or frequency, limiting downstream analyses.

How it works

The researchers formulate bird vocalization detection as an object detection task on spectrograms and train YOLO11 models to localize bird calls in dense tropical soundscapes from Singapore. This approach addresses the limitations of existing global-context methods that only indicate species presence without time or frequency localization, which is crucial for studying how animals adapt their vocalizations to acoustic environments.

The methodology involves several key steps:

1. Raw audio recordings are converted into spectrograms using the short-time Fourier transform (STFT).

2. Spectrograms are adapted for YOLO by selecting STFT parameters such that they contain approximately equal numbers of time and frequency bins, resulting in square images of size 1024 × 1024 after downsampling and padding.

3. Amplitude scaling is applied through a sequence: converting STFT magnitudes to log-power (decibel) values, clipping the spectrogram to the [1st, 99.8th] percentile range, and applying gamma compression with γ = 0.85 to enhance contrast in low-energy regions.

4. The single-channel spectrograms are converted into three RGB channels using the magma colormap to improve YOLO performance, as this mapping helps the model perform better on natural image statistics.

Annotation and Data

To facilitate analysis, an open-source browser-based annotation tool named BirdWatch was developed. Key features of this tool include:

** Time–frequency playback:**

The tool allows users to listen to time- and frequency-limited sections of recordings simply by drawing bounding boxes on the spectrograms, which is particularly useful in complex soundscapes with time-overlapping vocalizations.

** Bounding box annotation:**

Individual vocalizations can be annotated by drawing bounding boxes, and these annotations can be exported directly in YOLO format for model training. Furthermore, these boxes can be edited through a versatile set of tools allowing for quality checking, boundary refinement, and expert review.

The study utilized two datasets:

1. The Singapore Dataset (In-Distribution), collected across two recording sites (SBG1 and SBG2) with 18,095 bounding box annotations spanning 4 hours and 25 minutes of audio.

2. The Hawaii Dataset (Out-of-Distribution), used exclusively for out-of-distribution evaluation, containing 81,691 final labels after processing to align with the Singapore dataset's frequency range restriction (0.5–12 kHz).

Evaluation Metrics and Results

The researchers introduced Intersection over Minimum (IoMin) as an evaluation metric to better handle ambiguous acoustic boundaries than standard IoU, defined as:

** IoMin = intersection / min(areapred, areagt)**

The performance comparison against the unsupervised, energy-based Time-Frequency Event (TFE) detector showed that YOLO models significantly outperform the TFE detector baseline on both datasets and under both overlap criteria. On Singapore, the best-performing model achieved an IoMin@50 F1-score of 81.8%, nearly doubling the baseline's 42.1%.

The performance on Hawaii (OOD) was 57.6% under IoMin@50, compared to 48.6% for the TFE detector baseline, suggesting that YOLO appears to better suppress non-bird sources such as insect noise and human speech, producing tighter bounding boxes around individual calls.

Model Trade-offs and Future Directions

The study evaluated five standard-size variants of YOLO11 (nano to extra-large). While larger models showed a slight advantage on Hawaii, the YOLO11n model offered the most favorable performance–compute trade-off, falling within one percentage point of the best model on Singapore while requiring 10× fewer parameters and 13× fewer FLOPs.

Limitations identified include:

** Annotation boundary ambiguity:**

The paper notes that standard IoU can penalize predictions that only capture a target part partially, whereas IoMin yields one TP and zero FPs in such cases.

Future work suggests increasing the diversity of training data or constructing semi-synthetic soundscapes by overlaying isolated bird vocalizations onto real ambient noise recordings to gain finer control over species composition, call density, and SNR. Furthermore, exploring alternative input representations like Mel-STFT or wavelet transforms could potentially improve performance beyond the current log-STFT representation.

Conclusion

The paper presents a YOLO-based method to detect and localize bird vocalizations in both time and frequency from dense soundscapes, proposing IoMin as an alternative metric to IoU that better reflects detection quality for acoustic signals with ambiguous boundaries.

Here are the specific improvements for AI systems based on this research, and what these improved systems can achieve:

1. Acoustic Object Detection Preprocessing Module: The system can be used as a first-stage preprocessing module for long-duration soundscape analysis.

2. Time and Frequency Localization of Vocalizations: The system can precisely localize individual bird calls in both time and frequency within complex tropical soundscapes (e.g., Singapore).

3. Noise Rejection/Source Separation: The improved system can effectively ignore or suppress non-target sounds such as insect noise, human speech, and anthropogenic noise by producing tighter bounding boxes around bird calls compared to traditional energy-based methods (TFE detector).

4. Robust Evaluation of Acoustic Boundaries: By adopting the Intersection over Minimum (IoMin) metric instead of standard Intersection over Union (IoU), the system can be evaluated using a metric that better handles ambiguous acoustic boundaries, leading to more realistic performance assessments for time-frequency localization tasks.

5. Adaptive Soundscape Analysis: The ability to localize calls in time and frequency enables downstream analyses of how animals adapt their vocalizations in response to environmental changes (e.g., quantifying shifts in call timing or frequency based on noise source introduction).

6. Efficient Edge Deployment: Utilizing the YOLO11n architecture provides a high performance-compute trade-off, making the system suitable for low-power edge devices common in continuous, long-duration Passive Acoustic Monitoring (PAM) surveys.

Passive acoustic monitoring enables large-scale wildlife observation. Most bioacoustic classifiers predict species presence in a time window without localizing vocalizations precisely in time or frequency, limiting downstream analyses. We formulate time-frequency localization of bird calls as object detection on spectrograms and compare three computer vision model families (YOLO11, SAM 3, RF-DETR) against a non-learnable baseline. We introduce Intersection over Minimum (IoMin), an evaluation metric that better handles ambiguous acoustic boundaries than IoU. We also open-source a browser-based tool for efficient bounding-box labeling. The best RF-DETR model nearly doubles baseline performance on in-distribution, dense soundscapes from Singapore (83.4% vs. 42.1% IoMin@50 F1-score) and generalizes better to out-of-distribution recordings from Hawaii (63.2% vs. 48.6%). These results indicate that fine-tuned computer-vision models are well suited for time-frequency localization of bird vocalizations in complex soundscapes.