Link: http://arxiv.org/abs/2503.12874v1
PDF Link: http://arxiv.org/pdf/2503.12874v1
Summary: Large pre-trained vision-language models (VLMs), such as CLIP, demonstrateimpressive generalization but remain highly vulnerable to adversarial examples(AEs).
Previous work has explored robust text prompts through adversarialtraining, achieving some improvement in both robustness and generalization.
However, they primarily rely on singlegradient direction perturbations (e.
g.
,PGD) to generate AEs, which lack diversity, resulting in limited improvement inadversarial robustness.
To address these limitations, we propose anevolution-based region adversarial prompt tuning method called ER-APT, whichcombines gradient methods with genetic evolution to generate more diverse andchallenging AEs.
In each training iteration, we first generate AEs usingtraditional gradient-based methods.
Subsequently, a genetic evolution mechanismincorporating selection, mutation, and crossover is applied to optimize theAEs, ensuring a broader and more aggressive perturbation distribution.
The finalevolved AEs are used for prompt tuning, achieving region-based adversarialoptimization instead of conventional single-point adversarial prompt tuning.
Wealso propose a dynamic loss weighting method to adjust prompt learningefficiency for accuracy and robustness.
Experimental evaluations on variousbenchmark datasets demonstrate the superiority of our proposed method,outperforming stateof-the-art APT methods.
The code is released athttps://github.
com/jiaxiaojunQAQ/ER-APT.
Published on arXiv on: 2025-03-17T07:08:47Z