2.5-D Decomposition for LLM-Based Spatial Construction

The paper 2.5-D Decomposition for LLM-Based Spatial Construction addresses the critical challenge of enabling Large Language Models (LLMs) to perform complex, geometrically consistent spatial reasoning and construction tasks. While LLMs excel at symbolic manipulation and sequential planning, they often falter when required to maintain physical coherence or accurately model three-dimensional environments. This work introduces a novel framework that decomposes the high-dimensional task of 3D scene generation into manageable, lower-dimensional components—the 2.5-D space—thereby significantly improving the fidelity and robustness of LLM outputs for embodied intelligence applications.

The Limitations of Pure Textual Spatial Reasoning

The authors first establish that existing LLM approaches, even those utilizing advanced prompting techniques like Chain-of-Thought or Tree-of-Thought, struggle with inherent geometric constraints. They argue that treating spatial construction purely as a sequence of tokens leads to hallucinated geometry and inconsistencies in physical affordances. The paper notes that standard LLMs lack an internal mechanism to enforce Euclidean constraints, resulting in plans that are linguistically plausible but physically impossible. To overcome this, the authors propose moving beyond simple text-to-scene generation by explicitly decoupling the planning phase from the rendering and geometric validation phases.

The 2.5-D Decomposition Framework

The core contribution is the introduction of 2.5-D decomposition, which structures the problem space into three distinct, yet interconnected, modules: symbolic planning, depth mapping, and view synthesis. This decomposition allows the LLM to operate sequentially on increasingly constrained representations of reality. The framework posits that complex spatial tasks can be broken down into a series of manageable steps:

1. Symbolic Planning: The LLM first generates a high-level action sequence (e.g., move to the table, pick up the cup). This module handles the what and when of the task, acting as a traditional planner.

2. Depth Estimation and Constraint Mapping: The planned actions are then passed to a specialized module that estimates required depth maps and identifies geometric constraints, such as avoiding collision with obstacles or maintaining a stable grasp. This step grounds the symbolic plan in measurable physical space.

3. View Synthesis and Refinement: Finally, the constrained plan is used to guide a view synthesis model, which generates the final spatial output while ensuring that every generated element adheres to the constraints established in Step 2. The paper emphasizes that this iterative refinement loop drastically reduces geometric errors compared to end-to-end generation.

Integration with Embodied Agents and Control Policies

The framework is designed for direct integration with embodied AI systems, making it highly relevant for robotics and autonomous agents. The authors demonstrate how the decomposed output can be translated into actionable control policies, moving beyond mere visualization. They introduce a novel Constraint-Guided Policy Module which takes the predicted depth maps and geometric boundaries as input, ensuring that generated actions are not only logical but also physically executable in real-world environments.

The paper outlines several key technical advancements necessary for this integration:

• Geometric Grounding: Implementing a mechanism to translate abstract concepts (e.g., the largest object) into quantifiable bounding boxes and spatial coordinates.

• Multi-Modal Feedback Loop: Establishing a feedback loop where the predicted physical state is continuously validated against simulated sensor data, allowing the LLM to perform self-correction during planning.

• Hierarchical Task Decomposition: Structuring the prompt and reasoning process to handle tasks that require both fine-grained motor control and high-level goal setting.

In conclusion, by formalizing the separation between linguistic reasoning and geometric constraint satisfaction through 2.5-D decomposition, the paper provides a robust blueprint for building truly capable LLM agents that can reliably interact with and construct complex physical environments.

(Internal Monologue: The references provided detail a clear trajectory toward creating a truly autonomous, embodied, general-purpose AI agent. The gaps are not in theory but in integration, efficiency, and rigorous evaluation across all modalities. I must propose an architectural overhaul that synthesizes planning (symbolic), reasoning (LLM), and real-world action (robotics). This must be precise to avoid catastrophic failure.)

The current state-of-the-art models lack seamless integration between high-level symbolic reasoning, low-level physical control, and genuine self-correction in open environments. My proposed system is a modular architecture that forces the LLM to operate not just as a text generator, but as a System Orchestrator responsible for planning, execution monitoring, and error recovery.

• Improvement: Implementation of a multi-stage reasoning pipeline that mandates task decomposition before generating an action plan. This synthesizes the strengths of Plan-and-Solve Prompting [3] and Decomposed Prompting [4], while incorporating the self-reflection loop proposed in Inner Monologue [8].

• Mechanism: The system first receives a high-level goal (e.g., Build a functional prototype of X). It then executes three sequential internal modules:

1. Goal Decomposition Module (GDM): Breaks the goal into N discrete, ordered sub-tasks (T 1 to T 2 to).

2. Pre-computation Module (PCM): For each T i, it generates a detailed symbolic plan, often represented as executable code policies (leveraging Code as Policies [7]). This step explicitly integrates foundational knowledge from computational geometry [13] and classic vision processing models [12].

3. Self-Correction Module (SCM): After attempting T i, the SCM compares the actual observed state (from sensors) against the predicted state. If a discrepancy exceeds a defined threshold, the LLM is forced to generate an explicit root-cause analysis and propose an iterative correction plan, preventing cascading failures.

• Improvement: The system must move beyond mere language understanding to true affordance grounding. The LLM's output is no longer text; it is a structured JSON object containing the action, the target object, and the confidence metric. This directly addresses the gap highlighted by Do as I can, not as I say [6].

• Mechanism: The action space is constrained by a dynamic knowledge graph (KG) that maps linguistic concepts to physical capabilities (e.g., lift to requires grasping force F; open to requires torque T). When the LLM proposes an action, the system runs a physics simulation check against this KG before execution. This ensures that proposed actions are physically feasible and safe for the target environment (e.g., avoiding attempting to lift an object with insufficient payload capacity).

• Improvement: To achieve true autonomy in uncontrolled environments (like the open-world agents in [10] or [11]), the system must run efficiently on low-power, resource-constrained hardware. We must integrate advanced memory management techniques at the inference level.

• Mechanism: Utilizing PagedAttention [18] and deploying optimized, smaller model variants (similar to the principles of Nemotron [16]) on embedded platforms like Jetson Thor [17], we drastically reduce VRAM overhead and increase inference throughput. This allows the complex HDP planning cycle to run in near real-time on edge devices, making continuous, low-latency self-correction possible.

• Improvement: The system must be evaluated using a comprehensive benchmark that rigorously tests failure modes across modalities—a necessary step given the breadth of the literature [1].

• Mechanism: We institute a mandatory evaluation suite

Autonomous systems that build structures from natural-language instructions need reliable spatial reasoning, yet large language models (LLMs) make systematic coordinate errors when generating three-dimensional block placements. We present a neuro-symbolic pipeline based on 2.5-D decomposition: the LLM plans in the two-dimensional horizontal plane while a deterministic executor computes all vertical placements from column occupancy, eliminating an entire class of errors. On the Build What I Mean benchmark (160 rounds), GPT-4o-mini with this pipeline achieves 94.6% mean structural accuracy across 12 independent runs, within 3.0 percentage points of the 97.6% ceiling imposed by architect-agent errors that no builder-side improvement can address. This outperforms both GPT-4o at 90.3% and the best competing system at 76.3%. A controlled ablation confirms that 2.5-D decomposition is the dominant contributor, accounting for 28.7 percentage points of accuracy. The pipeline transfers directly to edge hardware: Nemotron-3 120B on an NVIDIA Jetson Thor AGX achieves 96.0% mean structural accuracy with the identical pipeline, slightly exceeding the cloud result. Expanding the system prompt by four targeted examples to exceed the model's 8,320-token prefix cache page size, combined with low-effort reasoning, reduces mean per-request latency by 3X to 19.7 seconds at 95.6% accuracy. The underlying principle, removing deterministic dimensions from the LLM's output space, applies to any autonomous construction or assembly task where gravity or other physical constraints fix one or more degrees of freedom. A transfer experiment on 500 IGLU collaborative building tasks confirms the effect generalizes beyond the primary benchmark.