CoLVR: Enhancing Exploratory Latent Visual Reasoning via Contrastive Optimization

arXiv:2605.08802 · cs.CV · Submitted 2026-05-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CoLVR: Enhancing Exploratory Latent Visual Reasoning via Contrastive Optimization".

Jane: The gist The proposed CoLVR framework introduces a latent contrastive training framework to enhance the exploratory capability of latent representations in Multimodal Large Language Models,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we just finished talking about how CoLVR tries to make these models explore more possibilities instead of just sticking to what they were trained on, and now let’s look at what the authors actually claim in this summary.

Jane: Right, so essentially, CoLVR is taking those latent representations—those hidden states inside the AI—and training them using a contrastive method. This means it’s not just forcing the state to match an image label; it’s actively pushing it away from things that look similar and toward things that are different in a meaningful way.

Lu: It builds positive samples based on result hints, and negative samples by adding angular perturbations to those latent states, using an InfoNCE loss to keep them balanced. That contrastive training is what creates this diverse latent space we talked about earlier.

Meng: So the core idea is that instead of having one fixed path for reasoning, the model gets rewarded for taking different, yet correct, paths in the hidden state space. It’s about creating a more flexible structure underneath the surface.

Tom: Exactly, and they show this isn't just theoretical stuff. They tested it on Visual Spatial Planning tasks and saw average improvements of five point eight three percent on VSP and eight point zero zero percent in Jigsaw task alone, which is pretty solid for performance gains <ref:2605.08802#pg3>

Jane: And what’s really interesting when I look at the results, Tom, is how they visualize this using UMAP projections. They show that CoLVR's latent tokens are spread out much more than other methods we looked at—substantially larger distances compared to Mirage’s clustering metric which was around sixteen point five four four <ref:2605.08802#pg2>

Lu: That distance in the UMAP space is the proof that it’s not collapsing into a single semantic manifold; it’s learning representations that have more distinct directions, which means they can handle more varied visual inputs.

Meng: From a deployment angle, this flexibility is what we want because it suggests better generalization. They even did some noise perturbation tests where CoLVR stayed stable under noise while other methods degraded severely, which speaks to robustness when things get messy in the real world <ref:2605.08802#pg3>

Tom: So it shifts the focus from just getting a score on one task to building a latent space that is inherently more exploratory and resilient, which is a significant mechanical change in how we train these systems.

Jane: It sounds like they’ve found a scalable way to encourage different reasoning behaviors by optimizing trajectories rather than just fixing the endpoints of those trajectories. But as the paper notes, it still uses image-question pairs with hints for its positive samples, so there's still that data cost involved.

Lu: Yeah, and they also flag that this shift to contrastive optimization makes the latent tokens less interpretable right now; they lack clear semantic grounding in their current form <ref:2605.08802#pg3>

Tom: So the paper argues that while it’s not perfectly clean for interpretation yet, the gain in flexibility and generalization across different benchmarks is substantial enough to warrant further exploration with things like probing heads to make those states more understandable later on.

The paper's summary: Tom: We just talked about how CoLVR uses contrastive training to encourage the latent space to be more diverse, and now let’s look at what that actually means for the model’s performance and reliability.

Jane: So, if we look at the improvements they show on those visual reasoning tasks—like Visual Spatial Planning—they see an average gain of five point eight three percent on VSP and eight point zero zero percent on the Jigsaw task <ref:2605.08802#pg3>

Lu: But the real thing is what they show in those UMAP visualizations, where CoLVR’s latent tokens have these much larger distances—ranging from about ten point five to twelve point six seven—compared to Mirage's clustering metric which was around sixteen point five four four <ref:2605.08802#pg2>

Meng: That distance difference is what matters because it proves the latent space isn't just a tight blob of similar things; it’s actually learning more distinct ways to represent visual concepts.

Tom: And that leads into the robustness part, because they tested this thing against noise and perturbation during reasoning, and CoLVR maintained stable performance under those conditions where other methods failed badly <ref:2605.08802#pg3>

Jane: That means the AI is learning representations that are more resilient when things get a little noisy or unexpected in a visual scene, which is exactly what you want for real-world use.

Lu: It’s about moving away from hard alignment objectives—where the latent space gets crushed into one shape—toward something that allows for more flexible adaptation to unseen scenarios.

Meng: So, this implies that when we deploy these models, we can trust them a bit more because they aren't just matching a fixed pattern; they have some real adaptability built into their hidden states.

Tom: Exactly. It’s about creating better latent tokens that are more flexible in adapting to those unseen visual scenarios, which is what the ablation studies confirm as the cornerstone of the whole framework.

Jane: The big picture here is that CoLVR moves us toward a system where the latent space isn't just a fixed map we draw ourselves, but something dynamic that actively explores different reasoning paths during inference. But they do admit one thing: it still needs those image-question pairs with result hints for its positive samples, which means there’s still some data cost to get that initial contrastive supervision set up.

The paper's improvements: Tom: So we’ve walked through the mechanics of CoLVR, seen how it improves performance on tasks like VSP, and looked at those UMAP visualizations showing better latent exploration, and now we’re coming to the final word on this paper.

Jane: The main thing to take away is that CoLVR moves us away from static feature matching toward contrastive trajectory optimization for enhancing exploratory capability in latent tokens. It’s a way to make sure the AI isn't just sticking to known paths but is genuinely exploring different reasoning behaviors in its hidden state space.

Lu: I think the authors successfully demonstrated that this approach prevents the latent space from collapsing into a homogeneous semantic manifold, which is something we desperately need when training models for complex visual tasks.

Meng: For deployment, it suggests a more flexible system that can adapt to unseen scenarios, and that robustness under noise is definitely a win for production stability.

Lalam: I’m confident in this answer; CoLVR introduces a more discriminative latent feature learning mechanism by rewarding unique hidden state trajectories during reinforcement learning.

Tom: It’s a solid framework, but they did admit it still relies on those image-question pairs with result hints for positive samples, so there's that data cost to keep in mind.

Jane: And they also flag that the shift to contrastive optimization trades off some interpretability; the resulting latent tokens don't have clear semantic grounding yet. That’s a limitation they point out directly.

Lu: That’s fair, and their future work hints at lightweight probing heads to project those exploratory states into a more human-understandable concept space later on.

Meng: Yeah, if we can get that interpretation layer on top of this exploration, that would make it much easier for us to debug what's actually going wrong when the model messes up at scale.

Tom: So, in short, CoLVR provides a scalable way to improve the flexibility and generalization of latent reasoning by encouraging more exploratory trajectories in the hidden state space. It’s a significant step toward building more resilient AI systems.

Jane: And that’s where we’ll leave it for this session today. Next up, we've got a paper on how vision-language models see more about agriculture than they report and what that means for test-time scaling.

Conclusion: Tom: So we’ve walked through the mechanics of CoLVR, seen how it improves performance on tasks like VSP, and looked at those UMAP visualizations showing better latent exploration, and now we're coming to the final word on this paper.

Meng: For deployment, it suggests a more flexible system that can adapt to unseen scenarios, and that robustness under noise is definitely a win for production stability.

Lalam: I’m confident in this answer; CoLVR introduces a more discriminative latent feature learning mechanism by rewarding unique hidden state trajectories during reinforcement learning.

Tom: It’s a solid framework, but they did admit it still relies on those image-question pairs with result hints for positive samples, so there's that data cost to keep in mind.

Lu: That’s fair, and their future work hints at lightweight probing heads to project those exploratory states into a more human-understandable concept space later on.

Meng: Yeah, if we can get that interpretation layer on top of this exploration, that would make it much easier for us to debug what's actually going wrong when the model messes up at scale.

Jane: And that’s where we'll leave it for this session today. Next up, we've got a paper on how vision-language models see more about agriculture than they report and what that means for test-time scaling.

Ziyang Ding, Linjian Meng, Yiming Wu, Yuhan Li, Yuhao Liu, žZhen Zhao

Shandong University · Shanghai AI Laboratory · Nanjing University · ÄThe University of Hong Kong

cs.CV

Submitted: 2026-05-09

Updated: 2026-10-19

Comments: Accepted by NeurIPS 2026. 17 pages, 10 figures, 9 tables

Code: https://github.com/Oscar-dzy/CoLVR

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The proposed CoLVR framework introduces a latent contrastive training framework to enhance the exploratory capability of latent representations in Multimodal Large Language Models, achieving

Key concepts

Latent Contrastive Training: This technique uses a contrastive objective during training to push the model's latent representations closer to desired outcomes (positive samples) while pushing them away from incorrect or structured negative samples. This forces the model to learn diverse and exploratory patterns rather than collapsing into a single, fixed representation.
Latent-Trajectory Contrastive Reward: This is a reward signal used during reinforcement learning that compares different sequences of latent states (trajectories). Instead of enforcing a specific final state, this reward encourages the model to select reasoning paths that are more informative and distinct from other possible paths, promoting exploration.
Latent Space Exploration: This refers to the ability of the model's internal hidden states (latent tokens) to cover a wide and varied area in its representation space. CoLVR achieves this by moving away from 'hard alignment,' which rigidly fixes these representations, allowing the model to discover more flexible and useful reasoning paths.
Angular Perturbations: These are controlled changes applied to latent representations, such as small rotations or shifts. By using these perturbations to create negative samples, the framework ensures that the model learns meaningful distinctions between similar visual concepts in a way that encourages broader exploration.

Terminology

Latent Visual Reasoning — 4× in this paper · explained in 2 episodes
Latent Representation — 2× in this paper · explained in 2 episodes
Latent Representations — 2× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Contrastive Optimization for Latent Visual Reasoning — 1× in this paper
Latent Reasoning — 1× in this paper · explained in 3 episodes
Latent Space — 1× in this paper · explained in 7 episodes
MLLMs — 1× in this paper · explained in 2 episodes
MLLMs (Multimodal Large Language Models) — 1× in this paper · explained in 2 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist The proposed CoLVR framework introduces a latent contrastive training framework to enhance the exploratory capability of latent representations in Multimodal Large Language Models, achieving significant performance improvements on visual reasoning tasks.

Problem and Motivation

Existing methods often rely on hard alignment objectives to force latent states to match predefined visual features, which severely limits the exploratory of latent reasoning process The paper identifies a fundamental limitation where hard alignment objectives constrain latent representations into a homogeneous latent space, thereby limiting the exploratory capability of latent tokens This constraint leads to fixed reasoning paths and reduces the diversity of latent representations. Furthermore, existing reinforcement learning methods often adopt predefined distribution assumptions and trajectory-level alignment objectives, which forces latent trajectories to match specific latent states observed during rollout instead of encouraging exploratory latent reasoning.

CoLVR Framework

To address these challenges, CoLVR introduces a latent contrastive training framework. This framework has two main components:

Latent Contrastive Supervised Fine-Tuning: After a warm-up stage, CoLVR constructs positive and negative samples via angle-based perturbations to introduce a contrastive objective in the latent space. Positive samples are defined as image data paired only with result hints, while negative samples are generated by applying angular perturbations to latent representations. The InfoNCE loss is used to optimize the latent representations, encouraging closeness to positive samples and separation from structured negative samples.
Latent-Trajectory Contrastive Reward: This objective is introduced in the reinforcement learning stage to provide relative preference signals among latent trajectories rather than enforcing predefined trajectory targets. The reward is defined case-wise, balancing exploration and discrimination by selecting a positive sample when correct answers are present or applying a repulsion constraint when no correct answer exists.

Experimental Results and Analysis

Extensive experiments demonstrate the effectiveness of CoLVR, achieving an average improvement of 5.83% on the VSP (Visual Spatial Planning) task and 8.00% in Jigsaw task. The model significantly outperforms constraint-based latent models across multiple out of domain benchmarks, with a 3.40% gain on MMStar. The UMAP visualizations confirm that CoLVR learns more exploratory latent representations, exhibiting substantially larger distances, ranging from approximately 10.5 to 12.67, compared to Mirage's lower clustering metric of 16.544. Furthermore, noise perturbation experiments show that CoLVR maintains stable performance under controlled perturbations while Mirage exhibits severe degradation, suggesting CoLVR learns more exploratory and robust latent representations.

Conclusion

In conclusion, CoLVR moves from static feature matching toward contrastive trajectory optimization to enhance the exploratory capability of latent tokens. This approach provides a scalable way to improve the flexibility and generalization of latent reasoning by encouraging more exploratory trajectories in the hidden state space. The method successfully prevents the latent space from collapsing into a homogeneous semantic manifold.

Limitations

The paper notes two limitations of CoLVR, first, it still relies on image-question pairs with result hints for positive samples, maintaining a non-trivial data cost. Second, the shift to contrastive optimization trades off interpretability for robustness; the resulting latent tokens lack clear semantic grounding. Future work could incorporate lightweight probing heads to project exploratory states into a human-understandable concept space.

Appendix Details

The detailed experimental settings specify that the latent tokens are consistently set to 8 during inference to maintain alignment between training and inference. The authors also adapted CoLVR’s inference protocol to match Mirage, finding that dynamically determining versus fixing the number of latent tokens has no measurable impact on CoLVR’s performance. The structured perturbation is critical for constructing meaningful latent contrastive supervision, as random Gaussian noise fails to provide the necessary semantic direction. The contrastive trajectory reward is crucial for guiding the model toward more flexible and exploratory reasoning paths by explicitly rewarding unique and informative hidden state trajectories.

Inference Examples

The paper provides inference examples for various tasks, including a VSP task where the action plan is RIGHT, UP, UP, RIGHT, and a Jigsaw task where the correct option is "C. These examples illustrate the model's ability to generate reasoning sequences in the latent space. The VSP example also shows an inference for a maze solver task resulting in the action plan RIGHT, UP, UP, RIGHT. The VisPuzzle example yields option B". The VMMVP example suggests the correct choice is (A) erect. The MMStar example indicates the largest body of water is at A: At the center of the image. The CV-Bench example shows that there are 1 pillows in the image (B).

Table Summary

Table 1 compares CoLVR with other VLMs, showing that CoLVR consistently outperforms baselines across multiple visual reasoning benchmarks, with overall improvements ranging from 1.34% to 8.00%. Table 2 further confirms the superiority of CoLVR on out-of-domain visual understanding benchmarks, where it consistently achieves the best performance across all evaluated datasets. Table 3 shows that removing the contrastive SFT and GRPO stages results in a 5.20% performance degradation in VSP seen scenarios. The ablation study confirms that the latent contrastive framework is a cornerstone for CoLVR training framework. The results suggest that latent tokens in CoLVR exhibit stronger flexibility in adapting to unseen visual scenarios. The final performance gain on the Jigsaw task is an 8.00% improvement over Mirage. The results highlight that CoLVR introduces a more discriminative latent feature learning mechanism. The paper concludes that CoLVR leads to a more exploratory and flexible latent space.

References

[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li. Qwen2.5-vl technical report

[13] Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Emad Barsoum, Muhao Chen, Zicheng Liu. Latent visual reasoning

[15] Leland McInnes, John Healy, James Melville. Umap: Uniform manifold approximation and projection for dimension reduction

[17] OpenAI. Gpt-4o system card

[19] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Deepseekmath

[25] Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang.

Improvements for AI systems

Bold latent space exploration via contrastive optimization: CoLVR introduces a latent contrastive training framework that learns diverse and exploratory representations with a latent contrastive objective guided by angle-based perturbation, which expands the semantic latent space and avoids over-constrained embedding. This allows the model to learn more exploratory latent representations compared to hard alignment methods that constrain tokens toward predefined features.
Reinforcement learning for fine-grained control: CoLVR employs a latent trajectory contrastive reward for RL post-training to enable fine-grained optimization of latent visual reasoning process and thus fostering diverse reasoning behaviors. This provides latent-level preference signals that encourage task-relevant latent rollouts during reinforcement learning, balancing exploration and discrimination.
Enhanced generalization on out-of-domain benchmarks: The method demonstrates improved transferability, showing CoLVR significantly improves the generalization ability of latent reasoning models and consistently achieves the best performance across all evaluated datasets compared to models relying on hard alignment constraints like Mirage.
Robustness to perturbation and noise: CoLVR is shown to be more robust under distribution shift; for instance, when injecting random noise into latent tokens during reasoning, CoLVR maintains stable performance, while Mirage exhibits severe degradation, suggesting it learns more exploratory and robust latent representations.