Discovering Global False Negatives On the Fly for Self-supervised Contrastive Learning

arXiv:2502.20612 · cs.LG, cs.AI, cs.CV, stat.ML · Submitted 2025-02-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Discovering Global False Negatives On the Fly for Self-supervised Contrastive Learning".

Jane: The gist: GLOFND is an optimization-based approach that automatically learns on the fly thresholds for each anchor data to identify its false negatives during training,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So looking at the whole paper, "Discovering Global False Negatives On the Fly for Self-supervised Contrastive Learning," it’s an optimization technique focused squarely on finding those tricky false negatives in self-supervised contrastive learning without needing massive batch sizes.

Jane: The authors show that by learning these per-anchor thresholds on the fly, they can improve how much information the model captures from its data, addressing issues where similar items are incorrectly separated during training.

Lu: It suggests that we don't have to rely on fixed rules for what constitutes a negative pair; instead, the system learns what’s right for each specific image anchor it encounters.

Meng: From a practical standpoint, this means our models could be more resilient when trained on smaller or standard-sized batches because the correction mechanism adapts to the current situation.

Lalam: This work points toward a future where representation learning is less about finding perfect pairs upfront and more about intelligently managing what we observe during the training process.

Tom: The paper does show that this method works well in unimodal, bimodal, and even semi-supervised contrastive learning setups with minimal extra computational overhead.

Jane: The main thing to take away is that if you're worried about those false negatives pushing your embeddings apart, GLOFND gives you a way to identify them globally and dynamically without slowing things down too much.

Conclusion: Tom: So, we’ve been talking about GLOFND, this method that learns thresholds for false negatives on the fly for self-supervised contrastive learning.

Jane: Right, it basically figures out which negative pairs are actually pulling our model away from what it should be seeing.

Tom: That's what the title says, "Discovering Global False Negatives On The Fly." It sounds a lot more active than just fixing a problem later.

Lu: It’s about dynamic filtering, Tom. Instead of guessing one fixed number for every image in the whole dataset, this system finds the right cutoff point for each anchor as it trains.

Meng: So, it adapts to the data structure during training, meaning if the batch looks weird today compared to yesterday, its filtering changes too. I’m curious how robust that actually is in a real deployment setting.

Jane: That adaptability is key because traditional methods often use static rules for what counts as a negative pair. This one lets the model learn the right sensitivity based on what it's currently processing.

Tom: And the results are pretty solid, showing improvements in identifying those false negatives over existing techniques like FNC. We saw gains like twenty percent in some tests on ImageNet100 for example.

Lalam: From my perspective, this means our representations become cleaner because we’re actively pruning the noise that confuses them during training. It helps build a more coherent understanding of visual data overall, which is a big step for culture and creativity.

Jane: It really does give us a better picture of what the model is actually seeing versus what it’s misinterpreting as irrelevant.

Tom: But we have to remember the authors did some important work on how this works across different learning styles, like unimodal or bimodal contrastive learning.

Lu: Exactly. They didn't just test one scenario; they showed this method works in several setups, which makes it much more versatile than something that only works in one specific context.

Meng: Versatility is good for engineering, but I still wonder about the computational cost when you have to run an optimization problem for every anchor during every training step. That’s a practical hurdle we need to watch closely.

Jane: That’s a fair point, Meng. The paper focuses on making this approach integrated with existing techniques with minimal overhead, which is what makes it more usable right now than some of the purely theoretical proposals out there.

Tom: So, GLOFND seems to be a practical way to improve contrastive learning by making it smarter about its own mistakes during training.

Lu: It’s a clever optimization-based approach that solves the false negative issue by dynamically setting global thresholds for each image anchor in the dataset. That's what’s really interesting here.

Jane: And that dynamic thresholding is what lets it handle those tricky semantic overlaps that usually mess up contrastive learning embeddings.

Tom: So, we’ve seen the method works across different learning styles and shows solid gains over established methods in a few key experiments.

Lalam: It's about improving how the AI learns to see and understand the world by making its training process more accurate on a fundamental level.

Texas A&M University

cs.LG, cs.AI, cs.CV, stat.ML

Submitted: 2025-02-28

Updated: 2025-06-25

Code: https://github.com/vibalcam/GloFND

Importance score: 82/100

The gist: The gist: GLOFND is an optimization-based approach that automatically learns on the fly thresholds for each anchor data to identify its false negatives during training, addressing a critical issue in

Key concepts

False Negatives
These occur when an anchor image and a sample from the dataset that should be considered a negative have similar meanings. In contrastive learning, this causes the model to incorrectly push these semantically similar items far apart in its embedding space, leading to poor representations.
GLOFND Algorithm
This is an optimization-based method that dynamically learns global thresholds for every anchor. It alternates between updating these per-anchor thresholds using SGD and then modifying the contrastive loss to exclude the identified false negatives, ensuring computation remains minibatch-wise.
Per-Anchor Thresholds ($\lambda_i$)
GLOFND maintains a unique threshold ($\lambda_i$) for each anchor ($x_i$) across the entire dataset. This allows the method to adapt to different definitions of false negatives, effectively filtering out only the most relevant false negatives specific to that particular anchor's context.
Optimization-Based Approach
The core idea is using convex optimization problems to determine these thresholds. The algorithm solves an optimization problem at each step to find a threshold that filters out the top-alpha percent of similarity scores, making it data-driven and adaptive rather than relying on fixed, pre-set rules.

Terminology

Summary

The gist: GLOFND is an optimization-based approach that automatically learns on the fly thresholds for each anchor data to identify its false negatives during training, addressing a critical issue in self-supervised contrastive learning where negative pairs with similar semantics are incorrectly pushed apart.

Problem and Motivation

Negative pairs in self-supervised contrastive learning can result in false negatives when an anchor image and a sample drawn from the entire dataset, excluding the anchor, have similar semantics, leading to their embeddings being falsely pushed apart <ref:2502.20612#pg2>. This phenomenon detrimentally impacts the representations learned through contrastive learning because it encourages the model to discard crucial semantic information <ref:2502.20612#pg4>. Previous approaches fall into local (batch-wise) and global (dataset-wise) categories, with local methods being unreliable when mini-batch sizes are small, and global methods like IFND being computationally expensive for large datasets due to clustering the entire dataset <ref:2502.20612#pg5>.

GLOFND Algorithm

GLOFND is introduced as a novel algorithm that learns global and dynamic thresholds for each anchor in the dataset. The algorithm alternates between two key steps:

  1. Updating the per-anchor thresholds by SGD to solve a convex optimization problem of finding a threshold that can filter out the top-α% of a set of scores.

  2. Updating the parameters of the encoder network by using a stochastic gradient estimator of the modified contrastive loss that takes care of the false negatives identified via the learned thresholds (e.g., excluding them).

The threshold update for an anchor xi in a mini-batch is computed using a stochastic estimator derived from solving an optimization problem where it casts the problem of finding the (1 − α)-quantile of all similarity scores as follows. The algorithm maintains a threshold λi for detecting global false negatives across the whole dataset for each anchor xi in D, while ensuring that computation remains minibatch-wise.

Integration and Dynamics

GLOFND can be integrated with various contrastive learning techniques with minimal computational overhead. The process involves updating the thresholds λi, i ∈ B, based on sampled negative data in the mini-batch, and then removing the identified false negatives from the loss function and updating the parameters w of the encoding network. For unimodal CL methods like SogCLR, GLOFND is combined with SogCLR by randomly sampling a batch B ⊂ D and data augmentations A, A′.

The threshold λi can be updated by calculating the stochastic subgradient of the optimization problem (2) and employing the regular SGD update. The parameter α allows GLOFND to adapt to different definitions of false negatives, which are inherently dependent on the desired level of granularity.

Experimental Results and Analysis

Experiments demonstrate the effectiveness of GLOFND in unimodal, bimodal, and semisupervised contrastive learning on several CL techniques without using a large batch size. In unimodal experiments on ImageNet100, GLOFND achieves significant improvements in false negative identification over FNC. For instance, GLOFND achieves a 20.83% and 5.14% improvement in false negative identification over FNC.

The ablation study verifies three aspects of GLOFND’s design: (i) the necessity to have a global threshold (as opposed to batch-wise), (ii) the necessity to have a different λi for each anchor xi ∈ D, and (iii) the quality of the learned λi threshold. The results show that GLOFND with per-anchor λi consistently outperforms the single-threshold variant. Furthermore, GLOFND approximates the desired threshold significantly better than FNC, achieving a Mean Absolute Error (MAE) of 0.1 and a Root Mean Squared Error (RMSE) of 0.13 (λi ∈ [−1, 1]), whereas FNC obtains MAE and RMSE of 0.21 and 0.28, respectively.

Conclusion

GLOFND successfully addresses the problem of identifying global false negatives in self-supervised contrastive learning through an optimization-based approach. Experimental results demonstrate that GLOFND improves existing contrastive learning methods, both for unimodal and bimodal tasks, with minimal computational overhead. The method is effective across different settings and shows statistical significance in improving performance in both semi-supervised and transfer learning scenarios.

Limitations

Since the focus of this paper is on false negative detection for contrastive learning, we address false negatives through filtering. Future work could explore more advanced methods that may further enhance downstream performance. The benefits of GLOFND and similar false-negative techniques on downstream tasks depend on the proportion of false negatives in the pretraining dataset and how false negatives are defined within the downstream task.

Acknowledgments

VB, BW and TY were partially supported by National Science Foundation Award 2306572 and 2147253, National Institutes of Health Award R01HL168116. CL was partially supported by National Institutes of Health Award R01HL168116.

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

References

Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Wallach, H., Larochelle, H., Beygelzimer, A., dAlche-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates Inc., 2019.

Bitton, Y., Bitton-Guetta, N., Yosef, R., Elovici, Y., Bansal, M., Stanovsky, G., and Schwartz, R. Winogavil: gamified association benchmark to challenge vision-and-language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022.

Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision, 2014.

Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep Clustering for Unsupervised Learning of Visual Features, March 2019. URL http://arxiv.org/abs/1807.05520.

Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations, June 2020a. URL http://arxiv.org/abs/2002.05709.

Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. Big Self-Supervised Models are Strong SemiSupervised Learners, October 2020b. URL http://arxiv.org/abs/2006.10029.

Chen, T.-S., Hung, W.-C., Tseng, H.-Y., Chien, S.-Y., and Yang, M.-H.

Improvements for AI systems

  1. Bold header: Automated False Negative Filtering for Contrastive Learning

GLOFND automatically learns on the fly the threshold for each anchor data to identify its false negatives during training, which allows for a dynamic adjustment of negative sample selection based on the current state of learned representations. This enables the system to select only those negative pairs that are truly dissimilar, as opposed to those with similar semantics that were incorrectly deemed negative.

  1. Bold header: Improved Representation Separation

By eliminating false negatives from the loss function, GLOFND achieves better representation quality; for instance, identifying and removing false negatives using GLOFND achieves better separation between the learned representations of different classes. This directly leads to better separation in the embedding space, which is empirically shown to result in improvements across linear evaluation and transfer learning tasks.

  1. Bold header: Scalable Global False Negative Discovery

GLOFND provides a global (dataset-wise) false-negative discovery approach that is agnostic to batch size and scalable for large-scale datasets. This means the system can reliably identify false negatives across the entire dataset without being constrained by small mini-batch sizes, unlike local methods which are limited by batch size.

  1. Bold header: Granularity Adaptability

The hyperparameter α allows GLOFND to adapt to different definitions of false negatives, which are inherently dependent on the desired level of granularity. This capability means the system can be tuned to align with specific semantic resolutions, such as distinguishing between coarse-grained task[s] of classifying between cars and animals versus fine-grained task[s] of classifying dog breeds.

Sources

Related papers