Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks

The paper Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks by Wojciech Zarzecki and Jarosław Arabas argues that existing global optimization benchmark suites are limited and proposes that Black-Box Adversarial Attack (BBAA) problems can serve as valuable, large-scale benchmarks for testing global optimization methods.

The authors state: "Existing global optimization benchmark suites are of a moderate size and are based on a small number of analytical functions that date back even to the 1970s. This causes a risk of biasing the development of global optimization methods. We argue that the tasks related to the black-box adversarial attack (BBAA) can serve as valuable global optimization benchmark in many-dimensional space."

The paper defines the BBAA as an optimization problem: We define the BBAA as an optimization problem that is aimed at finding the perturbation of the original image that results in misclassification and is similar to the noise with the smallest possible variance. The objective function is formulated as: L(δ) = − log pc(x) (x + δ) − α∥δ∥22, where c(x) is the ground-truth class, α is a hyperparameter controlling the importance of the L2 norm, and the feasible set is a hypercube [−ε, ε]n.

To demonstrate that the BBAA problem is multimodal, the authors introduce a local search method called Stochastic Growth Attack with Binary Search Refinement (SGA-BSR). They state: We formulate the local search method to demonstrate that the optimization problem is multimodal. Their experiments show that independent runs of SGA-BSR yield different alternative disturbance vectors δ, and that "for many images, the SGA-BSR yielded a variety of results which differed both in the objective function value and the perturbation magnitude. This evidences that the adversarial attack problem has many different local optima."

The paper then tests several global optimization methods on the BBAA problem. The methods compared include the classical Evolutionary Algorithm, four versions of Differential Evolution, Grey Wolf Optimizer, and the INFO (Efficient Optimizer based on Weighted Mean of Vectors) method. The experiments are conducted on CIFAR-10 and ImageNet datasets, with the attack targeting a VGG-like model for CIFAR-10 and a pre-trained Resnet-18 for ImageNet.

Key findings from the experiments include:

• The pixel perturbation strength ε is the most decisive factor for attack success on CIFAR-10: at ε = 0.01 every optimizer is effectively blocked, while at ε = 0.1 and ε = 0.2 most methods flip the majority of images.

• The regularization weight α has a non-monotonic effect, with a moderate value of α = 0.1 helping most optimizers.

• Across optimizers, INFO behaves like a greedy local search and plateaus at the lowest success rates, whereas DE, GEN, JADE, and SHADE spend more queries yet discover markedly stronger adversarial directions.

• The smallest perturbation strength is achieved by GEN and SHADE.

• For ImageNet, "low ε values make the attack much more difficult than for CIFAR-10, and no optimizer succeeds at ε = 0.01. On the other hand, at ε = 0.1, most attacks are successful, and several optimizers obtain even complete success at ε = 0.2."

• The relative ranking of methods remains stable across datasets: "GEN and SHADE dominate by achieving high success rates while keeping low disturbance strength, GWO performs worst whenever the objective lacks a regularization signal, and INFO stays the query-efficient but low-ceiling option."

The authors conclude: We demonstrated that the BBAA is a demanding global optimization task — it is multimodal, and the search space has very many dimensions. They plan future work to broaden the portfolio of optimization methods and to tune them to achieve better efficiency, extend the evaluation to targeted attacks, add more classifiers and datasets, and reformulate the objective function with a perceptual loss.

Improvements to AI Systems:

1. Adaptive Query-Budget Allocation in Black-Box Optimizers

• Improvement: Integrate the finding that DE, GEN, JADE, and SHADE outperform query-efficient methods like INFO in high-dimensional multimodal landscapes by dynamically switching between exploration and exploitation based on real-time success plateaus.

• Improved AI capability: An optimizer that automatically reallocates its query budget toward more diverse perturbation directions when it detects stagnation, leading to higher attack success rates at lower perturbation magnitudes without manual tuning.

2. Regularization-Aware Loss Shaping for Adversarial Perturbation Generation

• Improvement: Use the non-monotonic effect of α (optimal at 0.1) to design a meta-learning controller that adjusts the regularization weight during optimization, rather than keeping it fixed.

• Improved AI capability: A self-tuning adversarial attack system that avoids both over-regularization (which blocks attacks) and under-regularization (which yields large, detectable perturbations), producing stealthier and more transferable attacks across different ε values.

3. Multimodality-Aware Initialization for Image-Specific Attacks

• Improvement: Leverage the evidence that BBAA has many local optima (from SGA-BSR runs) to implement a multi-start strategy that seeds optimizers with diverse initial perturbations, then selects the best candidate.

• Improved AI capability: An attack generator that reliably finds multiple distinct adversarial examples per image, enabling robust adversarial training by exposing models to a wider variety of attack patterns, thus improving model robustness.

4. Dataset-Adaptive Perturbation Strength Calibration

• Improvement: Apply the finding that ε=0.01 blocks attacks on ImageNet but not CIFAR-10 to build a pre-screening mechanism that estimates dataset difficulty and automatically scales ε or switches optimization methods accordingly.

• Improved AI capability: A universal adversarial attack framework that works across diverse domains (e.g., medical imaging, autonomous driving) by predicting the minimal ε needed for success, reducing wasted queries and improving efficiency in real-time security applications.

5. Hybrid Optimizer Selection via Performance Prediction

• Improvement: Use the stable ranking (GEN/SHADE > DE/JADE > INFO > GWO) to train a meta-classifier that predicts which optimizer will perform best given image complexity, model architecture, and available query budget.

• Improved AI capability: An intelligent attack system that automatically selects the most effective optimizer per input, achieving near-optimal success rates with minimal computational overhead—useful for adversarial robustness evaluation of deployed AI models.

6. Perceptual-Loss Reformulation for Human-Invisible Perturbations

• Improvement: Replace the L2 norm in the objective with a perceptual similarity metric (as the authors plan), informed by the observed trade-off between perturbation magnitude and attack success.

• Improved AI capability: An adversarial attack that produces perturbations imperceptible to human observers while maintaining high misclassification rates, enabling safer testing of AI systems in sensitive applications like facial recognition or content moderation.

Existing global optimization benchmark suites are of a moderate size and are based on a small number of analytical functions that date back even to the 1970s. This causes a risk of biasing the development of global optimization methods. We argue that the tasks related to the black-box adversarial attack (BBAA) can serve as valuable global optimization benchmark in many-dimensional space. We demonstrate the efficiency of several types of evolutionary algorithms and other metaheuristics in solving example BBAA problems. Thus, we take a step towards convergence of global optimization methods to the challenges and needs that arise in the modern machine learning field.