Beyond Localization: A Comprehensive Benchmark of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images

Multimodal Large Language Models (MLLMs) show strong visual perception, yet remain limited in reasoning about space under changing viewpoints. The gist: PCSR-Bench reveals a clear gap where full-scene visual access improves coverage but does not yield robust viewpoint transformation, perspective re-anchoring, or compositional spatial reasoning.

The Problem and Motivation

The paper addresses the challenge of Perspective-Conditioned Spatial Reasoning (PCSR), defined as updating, re-anchoring, and inferring spatial relations under a changed or hypothesized observer position, orientation, and viewpoint. This capability is fundamental to spatial cognition but is often entangled with partial observability in current vision-language models. Existing benchmarks typically target fixed camera views and fail to directly isolate the core challenge of PCSR. To bridge this gap, the authors introduce PCSR-Bench, a diagnostic benchmark designed specifically to test whether models can recompute spatial relations under changed observer conditions.

PCSR-Bench Construction

The benchmark is built on 360° panoramic images and 3D scene annotations from the ReplicaPano dataset. A central design goal is to reduce the confound between incomplete observation and genuine reasoning failure by using these annotations to create a canonical 3D scene layout that unifies room geometry, camera pose, object instances, geometric extents, semantic categories, and instance identities in a single operable space. The benchmark is structured into three cognitive groups:

1. Perception (T0 Counting, T6 Egocentric Distortion Test the understanding of physical and geometric properties of 3D-to-2D panoramic projection).

2. Spatial (T1 Relative Distance, T2 Relative Direction).

3. Advanced PCSR (T3–T5), which probe whether models can update, re-anchor, and constrain spatial judgments under changed observer conditions.

Task Suite and Evaluation Metrics

PCSR-Bench is organized into eight diagnostic tasks (T0—T7) across three groups. The advanced PCSR tasks include:

(T3 Compositional Directional Chains)

(T4 Egocentric Rotation)

(T5 Perspective Re-anchoring)

The evaluation protocol employs a Semantic-Aligned Evaluation Scorer M to define task-level accuracy, which can be strict (exact match) or relaxed (allowing for intent canonicalization, synonym expansion, instanceto-class relaxation). The results show a substantial perception–reasoning gap, with accuracy dropping from 57.59% on Limited Field-of-View Reasoning (T7) to as low as 0.64% on open-ended Compositional Directional Chains (T3).

Diagnostic Findings and RL Intervention

Systematic zero-shot evaluation reveals that models perform nontrivially on foundational perception tasks, but degrade sharply on advanced PCSR tasks. To probe the plasticity of this gap, the authors conduct an RL-based diagnostic study using reward shaping. The results show that suitable reward design and task feedback improve performance on some advanced PCSR, suggesting that PCSR exhibits partial plasticity rather than being fully immutable. However, these gains are task-selective and sensitive to the evaluation protocol.

Methodological Implications

The study concludes that PCSR is a genuine bottleneck with limited but non-negligible recoverability. Key takeaways include:

1. A clear decoupling between model scale and performance on PCSR-Bench, as improvements in design can surpass, or even surpass, gains from scale.

2. The necessity of explicit modeling and targeted training to achieve robust spatial reasoning.

3. The high protocol sensitivity of the benchmark conclusions; factors like inference backend, visual preprocessing, and answer parsing rules materially shape the final benchmark conclusions.

Egocentric Rotation Case Study

A qualitative case study illustrates this gap: for a 90-degree counter-clockwise rotation, the authors' method correctly predicts Door, whereas a baseline model incorrectly predicts window. This demonstrates that RL-optimized methods maintain a coherent spatial update throughout its trace and produce a final answer (door) that perfectly aligns with its intermediate reasoning. This qualitative evidence confirms that the reward mechanism successfully penalizes disjointed logic and enforces strict consistency between the spatial reasoning chain and the final prediction.

Conclusion

The findings show that strong perceptual coverage does not guarantee robust spatial modeling; current MLLMs struggle to simulate spatial changes, treating relations as static 2D correlations rather than dynamic 3D structures. Further progress requires "stronger spatially grounded modeling, more effective reward design, and more carefully controlled evaluation protocols.

Here are specific improvements for AI systems based on the findings of this paper, categorized by technical focus:

) 1. Enhance Perspective-Conditioned Spatial Reasoning (PCSR) via Explicit Re-anchoring Mechanisms

The core weakness identified is the model's inability to re-anchor spatial relations under changed observer conditions (T5 PRA).

• Create a dedicated module or fine-tuning objective that explicitly trains models to perform viewpoint transformation and coordinate system re-anchoring, rather than relying on implicit 2D/360° projection heuristics.

• Implement an Observer State Encoder within the MLLM architecture that generates a latent representation of the current observer's pose (position and orientation). This state encoder must be explicitly conditioned into the spatial reasoning backbone during inference, forcing the model to treat spatial queries relative to this dynamic state.

) 2. Develop Task-Selective Reasoning Modules for Compositional Chains

The paper shows a critical disparity: models perform well on transitive compositional chains (T3 Transitive) but fail catastrophically on open-ended/compositional chains (T3 Open-ended).

• Implement a specialized reasoning pathway for compositional tasks that prioritizes multi-step logical deduction and symbolic manipulation over pure pattern matching. This could involve integrating a lightweight, external symbolic reasoning engine or a specialized attention mechanism designed to track relational dependencies across multiple steps.

) 3. Integrate Reward-Guided Reinforcement Learning (RL) for Reasoning Consistency

The RL study proves that targeted reward design can improve internal logical consistency, reducing the gap between intermediate thought and final output (as seen in the Egocentric Rotation case study).

• Develop a structured reward function that heavily penalizes reasoning–answer mismatch by requiring alignment between the generated intermediate reasoning trace (the "" block) and the final answer token sequence. This moves beyond simple accuracy metrics to enforce logical coherence.

• Use RL fine-tuning not just for aggregate score maximization, but specifically to optimize for Reasoning Coherence, ensuring that the model's internal spatial model is consistent throughout a complex transformation (e.g., 90° rotation).

) 4. Implement Protocol-Aware Inference Tuning

The results strongly indicate that benchmark conclusions are highly sensitive to the inference protocol (backend, budget, parsing rules).

• Create an automated Protocol Sensitivity Monitor for MLLMs. Before deployment on a new evaluation stack, this monitor would run a small suite of critical PCSR benchmarks and report performance variance across different inference settings (e.g., varying context window size or decoding strategies). This allows engineers to know if a model's success is due to inherent capability or protocol bias.

) 5. Optimize Model Architecture for Spatial Grounding

Since the bottleneck is treating spatial relations as static 2D correlations rather than dynamic 3D structures, architectural changes are warranted.

• Investigate integrating explicit 3D geometric priors (e.g., learned occupancy grids or implicit surface representations) directly into the Vision encoder's intermediate layers. This would give the model a better foundation for performing true Euclidean and rotational transformations, mitigating reliance on image-plane heuristics derived from panoramic projection distortions (T6).

The improved AI system can now perform:

1. Perform robust, viewpoint-invariant spatial reasoning by explicitly modeling and conditioning the observer's pose during inference.

2. Execute complex, multi-step spatial transformations with high logical consistency, as verified by RL training on coherence rewards.

3. Diagnose its own performance limitations against known evaluation protocols to ensure reliable deployment in real-world scenarios (Protocol Awareness).