Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping

The gist: Inverse-LLaVA proposes a multimodal architecture that inverts conventional alignment by projecting text embeddings into continuous visual representation space for fusion within intermediate transformer layers, eliminating the need for explicit alignment pretraining.

Introduction and Motivation

Traditional multimodal learning approaches typically rely on alignment pre-training to bridge vision and language modalities by projecting visual features into discrete text token spaces using large-scale image–text data The standard approach involves a computationally expensive two-stage training process where continuous visual features undergo alignment pre-training to conform to the discrete structure of text embeddings, which may limit the preservation of fine-grained spatial and photometric detail important for visual understanding This conventional wisdom rests on an assumption that may be unnecessarily constraining because it relies on a computationally expensive two-stage training process involving alignment pre-training and instruction tuning

Inverse Mapping Strategy

The core of Inverse-LLaVA is the proposal of mapping text embeddings into the richer continuous visual space instead of constraining continuous visual features to conform to discrete text distributions This architectural inversion preserves the continuous nature of visual information while maintaining compatibility with existing language model architectures The proposed mapping is formulated as Tproj = Wt2vT + bt2v, where T ∈ R dh×n are the text embeddings and Wt2v ∈ R dv×dh projects those text embeddings into visual feature space This design preserves visual continuity while leveraging the model’s continuous internal representations, removing the need for alignment pre-training and its associated representational constraints

Vision-Text Fusion Mechanism

Inverse-LLaVA employs a vision–text fusion design that departs from LoRA's low-rank update philosophy by introducing selective additive fusion modules to dynamically fuse visual and textual embeddings For a selected subset of attention layers l ∈ S ⊆ the set of layers, the fusion is applied by augmenting the standard query, key, and value projections with vision-fused components Specifically, the equations show how Ql, Kl, and Vl are computed by concatenating W(l)t2vHl along with Vemb The text-to-vision projection W(l)t2v maps from the language model’s hidden dimension dh to the visual dimension dv, creating a more compact yet expressive visual-aligned representation space

Training Objective and Advantages

To optimize the model, Inverse-LLaVA adopts a unified end-to-end training scheme that minimizes the standard autoregressive language modeling loss L = −T Xtarget i=1 log P(t i I, t<i; θ) This eliminates the alignment stage entirely and enables instruction-only training of lightweight fusion modules under identical backbones and data for controlled, apple-to-apple comparison with alignment-based frameworks The key advantages of this inverse approach include preserving visual continuity, eliminating alignment cost, achieving parameter efficiency by training only lightweight projection matrices and scaling parameters, and enabling adaptive fusion through learnable scaling parameters α

Experimental Results and Implications

Across nine multimodal benchmarks, Inverse-LLaVA demonstrates strong learning efficiency under reduced supervision, achieving substantial gains on reasoning-intensive tasks while exhibiting selective performance drops on perception tasks that depend on explicit visual–text grounding The results reveal that the inverse mapping strategy achieves competitive performance across nine vision–language benchmarks while fundamentally rethinking cross-modal interaction Specifically, Inverse-LLaVA outperforms LLaVA-1.5 on MMVET (31.2 vs 31.1), VizWiz (50.95 vs 50.0), and on ScienceQA-IMG (67.84 vs 66.80) The performance patterns suggest that the approach excels at reasoning tasks while struggling with direct visual-text correspondence, validating the hypothesis that inverse mapping favors abstract reasoning over direct visual-text correspondence This design decouples representation structure from supervision regime, suggesting that architectural innovation can substitute for data scale The findings motivate a new direction for multimodal architecture design that decouples representation structure from supervision regime

Conclusion

Inverse-LLaVA presents a multimodal architecture that inverts the conventional design by projecting text embeddings into continuous visual space rather than constraining visual features to discrete textual representations This representation-first design enables effective multimodal reasoning without relying on an explicit alignment pretraining stage The results show that preserving continuous visual signals in separate processing dimensions substantially benefits reasoning-oriented tasks, as evidenced by large gains in Numerical Calculation (+69.2%) and Text Translation (+125%) At the same time, correspondence-oriented tasks exhibit selective performance gaps that our analysis attributes primarily to the absence of paired supervision rather than architectural constraints This dichotomy highlights the role of supervision regime in shaping multimodal capabilities and suggests that different task categories place distinct demands on representation and training The findings motivate a new direction for multimodal architecture design that decouples representation structure from supervision regime

Declarations

No new data were created during this study The code used in this study is publicly available at https://github.com/xuhuizhan5/Inverse-LLaVA The authors declare that they have no competing interests This research is supported in part by the National Science Foundation (NSF) under grant number IIS2524380 and a Compute Grant from the Data Science Institute at Vanderbilt University Xuhui Zhan contributed to Conceptualization, Methodology, Software, Investigation, Formal analysis, and Writing–original draft Tyler Derr contributed to Conceptualization, Methodology, Supervision, Writing–review & editing and Funding acquisition All authors read and approved the final manuscript. Xuhui Zhan's e-mail is xuhui.zhan@vanderbilt.edu and Tyler Derr's is tyler.derr@vanderbilt.edu. The paper was submitted to arXiv on 15 Jul 2026. The paper is identified as arXiv:2508.12466v2 [cs.CV]. The paper is available on page 17 The funding acquisition details are supported by the National Science Foundation (NSF) under grant number IIS2524380 and a Compute Grant from the Data Science Institute at Vanderbilt University. The authors would like to thank the Data Science Institute at Vanderbilt University for providing computational resources and support. The references section contains numerous citations including Flamingo Alayrac et al (2022) and LLaVA Liu et al (2023, 2024a). The paper discusses the continuous nature of representations, citing Marro et al. (2025). The paper compares Inverse-LLaVA with LLaVA-1.5 and other models across nine multimodal benchmarks. The performance comparison table shows Inverse-LLaVA achieving 67.84 on MMEp. The analysis suggests that architectural design can substitute for data-intensive alignment procedures. The paper discusses the representational bias hypothesis, positing that observed patterns arise from consistent biases in how information is internally represented and processed. The paper concludes by suggesting that architectures should prioritize preserving the natural characteristics of each modality rather than forcing alignment with text.

1. Inverse-LLaVA projects text embeddings into continuous visual space and performs fusion within intermediate transformer layers, which preserves visual continuity while leveraging the model’s continuous internal representations. This allows for multimodal reasoning without relying on an explicit alignment pretraining stage, as it eliminates the need for costly alignment pretraining while preserving the expressive power of continuous visual representations.

2. The architecture achieves 'strong learning efficiency under reduced supervision', specifically showing substantial gains on reasoning-intensive tasks while exhibiting selective performance drops on perception tasks that depend on explicit visual–text grounding. This indicates the improved system will excel at reasoning-oriented evaluations, such as Numerical Calculation (+69.2%) and Text Translation (+125%), which are bolstered by the preserved continuous visual signals.

3. The design decouples representation structure from supervision regime, leading to a system that decouples architectural design from supervision regime, enabling alignment-free training by default while remaining extensible to richer supervision when needed. This allows researchers to prioritize reasoning over continuous visual structure and operate effectively even when such paired supervision is absent.