ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

summary

The gist

The gist: ZOMP proposes a query-efficient, fully forward-only method that tunes deep prompts in both vision and text branches of frozen CLIP models using simultaneous perturbation stochastic

In short

ZOMP is a forward-only method to tune deep prompts in vision-language models without needing backpropagation through frozen encoders. It achieves high performance under limited query budgets by combining cross-modal low-rank factorization, gradient-correction momentum, and a budget-indexed rank schedule. This allows it to efficiently explore the search space and find state-of-the-art results.

Key concepts

Cross-Modal Low-Rank Prompt Parameterization: This technique links the vision and text prompts at every layer using a shared factor. Instead of training separate parameters for each modality, ZOMP creates prompts as the product of a small shared vector and a modality-specific vector. This significantly reduces the total number of trainable variables needed to tune deep prompts.
Simultaneous Perturbation Stochastic Approximation (SPSA): SPSA is an optimization method used here to estimate gradients without needing full backpropagation. It works by evaluating the loss function at random, symmetric offsets from the current parameter values. This provides a noisy but computationally cheap estimate of the gradient direction for updating the model parameters.
Budget-Indexed Rank Schedule: This schedule controls how much of the prompt's complexity is explored based on how many training queries are left. Initially, it restricts perturbations to only a small number of low-rank components. As more queries are spent, it gradually unlocks higher-rank components, allowing the model to explore a larger search space when needed.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Flow Matching — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 17 episodes

This episode discusses

The paper

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models · Read on arXiv

Electrical and Computer Engineering Department, University of California Santa Barbara · Computer Science Department, University of California Santa Barbara

Prompt tuning adapts vision--language models with few trainable parameters, but computing their gradients still requires backpropagation through frozen encoders. We introduce ZOMP, a forward-only method for deep multimodal prompt tuning under a limited loss-query budget. ZOMP combines shared prompt factors with a budget-indexed rank schedule and gradient-correction momentum. Early updates search a rank-one subspace; later updates activate additional components without discarding the existing optimizer state. With 5,000 training-loss queries, ZOMP achieves 68.2% average 16-shot accuracy across 13 datasets, improving on ZIP by 6.0 points, alongside gains in base-to-new and distribution-shift generalization. On five image--text retrieval datasets, it outperforms four derivative-free baselines at both training sizes, improving average bidirectional Recall@1 over BPT-VLM by 2.63 points. An exploratory five-benchmark language diagnostic comparison examines caption discrimination beyond the retrieval galleries. Component, modality, depth, and budget studies characterize the method's benefits. ZOMP requires an internal-prompt inference interface and reduces allocated training memory relative to a matched-architecture first-order reference.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models".

Jane: The gist: ZOMP proposes a query-efficient,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper called "ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models," and the main thing they're saying is that fine-tuning vision models like CLIP usually needs backpropagation through the whole model, which is just not possible when you only have access to forward passes.

Jane: Right, so it addresses that exact problem because prior methods avoid backpropagation but often tune prompts in only one modality or search spaces so big they need thousands of forward passes, which just isn't practical with real query budgets.

Lu: The thesis here is proposing ZOMP as a query-efficient, fully forward-only method that tunes deep prompts in both the vision and text branches of a frozen CLIP model using simultaneous perturbation stochastic approximation.

Meng: What does that mean for us on the production side? It means they're trying to find a way to tune these huge models without needing those massive backpropagation steps we can't afford on our edge devices or in deployment.

Lalam: The core idea is combining three specific ingredients: a cross-modal low-rank reparameterization, which ties the two branches through a shared factor, a budget-indexed rank schedule to control how much they tune, and gradient-correction momentum to stabilize the noisy estimates.

Tom: That sounds like they're trying to manage that search space by making it small initially and then widening it only when the optimizer finds a good direction.

Jane: Exactly, because they argue that instead of trading expressiveness for a smaller search space, ZOMP restructures the prompt itself using low-rank factorization.

Lu: They achieve this through a cross-modal low-rank factorization where the vision and text prompts at each layer are generated as the product of a small shared factor and a modality specific factor.

Meng: So, they're creating this shared factor that links both branches so that one update moves both coherently, which is a really clever way to handle the complexity.

Lalam: And because component k occupies its own coordinates in the trainable vector theta, it lets them control which parts of the rank schedule affect first.

Conclusion: Tom: So, looking at "ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models," the authors are essentially showing that you can tune deep prompts in both vision and text branches of a frozen CLIP model without ever doing backpropagation.

Jane: The implication here is that this method makes deep multimodal prompt tuning practical under realistic query budgets, which is a big deal because it unlocks accuracy levels previously only available when backpropagation was used.

Lu: The authors show that the two mechanisms they introduced—the low-rank factorization and the budget-indexed schedule—are complementary; neither one works well on its own.

Meng: They also show that ZOMP reaches its target accuracy in the fewest queries to target of any backpropagation-free method, even though it has more trainable parameters than some other methods.

Lalam: The paper concludes that jointly exploiting multimodality and low-rank structure is an effective route to practical, query efficient prompt tuning, and they show that this approach generalizes better than prior work across base-to-new transfer and outofdistribution settings.

Tom: So, it boils down to structuring the search space via cross-modal low-rank factorization and stabilizing the noisy SPSA estimate with gradient-correction momentum.

Jane: That's what makes it a viable method for deep vision prompting without needing backpropagation, and they do this under a realistic query budget of five thousand training loss queries.

More episodes

← Home