GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

arXiv:2609.39601 · cs.CV, cs.AI, cs.LG, cs.RO · Submitted 2026-09-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives".

Jane: As a fastidious and diligent researcher, I have meticulously analyzed both provided texts regarding "GroundingPI:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title of this paper, "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives," and who came up with it. This title tells us immediately that the core idea revolves around grounding, which is connecting concepts to physical locations in an image.

Jane: And those authors—Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang... they seem like a solid team with deep expertise across different areas of vision and modeling. Their collaboration suggests they've put together a really comprehensive approach to this grounding problem.

Lu: The authors have clearly put together something that bridges the gap between the visual understanding models we see in general and the specific spatial reasoning needed for physical intelligence, which is a very ambitious goal. It shows how researchers are trying to make AI systems more physically aware.

Meng: I'm interested in how they structured their approach; having that many contributors suggests they tackled multiple facets of this problem, from the initial multimodal pretraining all the way through the reinforcement learning part for action alignment. That kind of breadth is usually a sign of a very thorough effort.

Lalam: Having a large group of authors working on this points to how many different AI ideas are converging right now; it shows that grounding isn't just one single technique, but something requiring deep integration across vision, language, and reinforcement learning. It’s pretty impressive to see that level of collaboration.

The paper's summary: Tom: So, summarizing the main points of "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives," the paper introduces a 4B parameter grounding foundation model designed specifically to generate precise spatial coordinates as points and boxes using a shared vocabulary. This model is trained using multimodal pretraining, supervised fine-tuning, and reinforcement learning with GRPO, all aiming to make it better at physical tasks than existing vision-language action models.

Jane: That’s the core idea explained simply: instead of just seeing an object and knowing its name, this AI learns to specify exactly where that object is located in a coordinate system. It achieves this by training on visual and language data, then fine-tuning it specifically so it can output these coordinates fast enough for control systems.

Lu: The summary emphasizes that the model’s strength comes from learning robust spatial relationships rather than just surface-level object recognition; they are building a foundation based on visual primitives that capture geometry and location very well.

Meng: I see the training regimen as key here; combining multimodal and spatial pretraining, supervised fine-tuning, and then reinforcement learning with GRPO is a sophisticated way to ensure the model learns not just what things look like, but how to use that knowledge for actual physical actions.

Lalam: It’s really about making AI more useful because this grounding capability allows the system to connect its high-level understanding of language directly to physical placement in a scene, which is a significant step forward for practical applications.

The paper's improvements: Tom: Now let’s look at what they actually improved. They show that GroundingPI establishes a new state of the art across thirty-four grounding benchmarks, averaging seventy-three point six eight percent, which is above models like GPT-six Astra, and it shows significant transferability to real-world physical tasks where it beats mainstream backbones by up to twenty-four point eight percent in some out-of-distribution settings.

Jane: That performance metric is quite telling; seeing that seventy-three point six eight percent average on those benchmarks shows it’s performing exceptionally well when compared against the other models they tested, especially in driving and robotics where precision is essential for success.

Lu: The paper highlights that dense grounding provides the strongest spatial core for manipulation and driving tasks, and that combining this with auxiliary perceptions like OCR or layout estimation gives the best results across both manipulation and driving scenarios.

Meng: Practically speaking, those improvements mean we can expect systems to be much more reliable when they are operating in environments they haven't seen before because the model has a better sense of spatial relationships than previous models.

Lalam: I think this means that future AI systems will be able to handle more complex physical tasks with less training data, which is really important for deploying AI in varied and unpredictable real-world settings.

Conclusion: Tom: So, wrapping up our discussion on "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives," the main implication is that we now have a foundation model that can reliably map instructions to precise physical locations, which is what truly enables reliable physical interaction for AI.

Jane: That capability means that as we move toward more sophisticated robots and autonomous systems, they won't just be guessing where things are; they'll have a much stronger internal sense of the scene geometry thanks to this model.

Lu: I think the long-term potential is huge because it suggests a clear path for developing AI that can handle complex, unstructured physical interactions by grounding every action in precise spatial awareness.

Meng: From an engineering viewpoint, it’s exciting because it means we might be able to build robots that are far more dexterous and capable of handling unexpected situations without needing massive amounts of perfectly labeled data for every single new scenario.

Lalam: I feel that this work sets a strong direction for how we should think about building AI systems—focusing on this spatial intelligence as a primary requirement, which could really improve how we design the next generation of helpful and physical AI tools.

Qize Yu, *Lianrui Fan*, *Boyu Chen*, *Jiaqi Liang*, *Xini Ding*, *Yue Chen*, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song8, Bohan Zhou2, Mingleyang Li2

XPeng Inc. · *Peking University* · *The University of Hong Kong* · *University of California, Berkeley* · *Princeton University* · *National University of Singapore* · *Tsinghua University*, '*HKUST (GZ)

cs.CV, cs.AI, cs.LG, cs.RO

Submitted: 2026-09-30

Updated: 2026-09-30

Comments: 64 pages, including supplementary material. Project page: https://groundingpi.github.io/ Code: https://github.com/groundingpi/GroundingPI Model: https://huggingface.co/GroundingPI/GroundingPI

Code: https://github.com/groundingpi/GroundingPI

Project page: https://groundingpi.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: As a fastidious and diligent researcher, I have meticulously analyzed both provided texts regarding "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives."

Key concepts

Visual Primitives
These are the fundamental building blocks of the model's knowledge. Instead of just recognizing objects, the model learns to represent physical concepts like 'point' or 'box' as encoded spatial coordinates within a shared vocabulary. This allows it to understand where things are in a scene with high precision, which is essential for physical interaction.
Dense Grounding
This refers to the model's ability to generate numerous, overlapping spatial points and boxes across an image. Dense grounding provides the strongest spatial core for manipulation and driving tasks because it offers a richer, more detailed representation of physical relationships compared to sparse or single-point grounding.
GRPO (Generalized Policy Optimization)
This is the reinforcement learning algorithm used during training to align the model's perceptual outputs with successful physical actions. GRPO helps guide the model's spatial predictions toward generating outputs that lead to actual, successful physical outcomes in a real-world environment.
Physical Intelligence Transfer
This measures how well GroundingPI performs on new, unseen physical tasks compared to other models. High transferability shows that the model has learned robust, generalizable rules for spatial reasoning and interaction, making it effective when applied to novel physical scenarios.

Terminology

Summary

As a fastidious and diligent researcher, I have meticulously analyzed both provided texts regarding GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives. The combination of these excerpts reveals a comprehensive picture of this work, detailing its architecture, training methodology, performance metrics across various benchmarks, and the core theoretical contributions.

Here is a detailed and comprehensive summary synthesizing the information from both sources:

GroundingPI is introduced as a 4B-parameter grounding foundation model specifically designed to bridge the gap between general semantic understanding and reliable physical interaction. Its core innovation lies in its ability to generate precise spatial coordinates—points and boxes—encoded within a shared vocabulary, effectively acting as a perceptual interface for physical intelligence tasks.

The model's capability stems from a sophisticated training regimen that combines multiple pretraining strategies:

  1. Multimodal and Spatial Pretraining: The foundation is built upon both multimodal data (visual and language) and spatial data, establishing a strong grounding base.

  2. Supervised Fine-Tuning (SFT): This refines the model's ability to map visual inputs to grounded outputs.

  3. Reinforcement Learning with GRPO: The training incorporates reinforcement learning using the Generalized Policy Optimization (GRPO) algorithm, which is crucial for aligning the model's perceptual outputs with successful physical actions.

The paper explicitly frames GroundingPI as a foundation model built on visual primitives, suggesting that its strength lies in learning robust spatial relationships rather than just surface-level object recognition.

GroundingPI demonstrates exceptional performance across a broad spectrum of grounding and perception tasks, establishing new state-of-the-art results:

  • Overall Grounding: GroundingPI achieves an average score of 73.68% across 34 grounding benchmarks, surpassing 44 baselines and leading leading Vision-Language Action (VLA) and World Action Model (WAM) models in driving and robotics. This performance is notably superior to the larger GPT-6 Astra model, which scored 71.54%.

  • Physical Intelligence Transfer: The model shows significant transferability to real-world physical tasks. On RoboTwin 2.0, GroundingPI consistently outperforms every mainstream backbone in all four out-of-distribution (OOD) settings by up to 24.8% relative to the strongest baseline. Similarly, on RoboCasa-GR1, it outperforms baselines trained with 75% of the demonstrations compared to those trained with 50%.

  • Specific Metrics: GroundingPI achieves high scores on point localization tasks (e.g., 84.76 F1@Point on HumanRef and 90.26/90.06 F1@Point on RefCOCOg validation/test), and strong performance in unseen data splits, showing minimal degradation between familiar and unseen splits (e.g., 76.00/75.00 on RefSpatial Location/Placement).

  • Auxiliary Perception: The model excels when combined with auxiliary perception tasks like OCR, layout estimation, and GUI understanding. For instance, it achieves an 85.72 F1mIoU on DocLayNet, outperforming both DocLayout-YOLO and Astra by a substantial margin (81.10 and 77.54, respectively).

The research identifies several critical insights that define GroundingPI's impact:

1. Perceptual Foundation for Physical Intelligence:

The central thesis is that reliable physical interaction requires connecting instructions to precise spatial targets, a capability general semantic understanding alone cannot guarantee, especially when action supervision is scarce. GroundingPI directly addresses this perceptual gap by providing a strong, precise spatial core.

2. The Value of Dense Grounding and Composition:

  • Spatial Core Strength: Dense grounding provides the strongest spatial core in leave-one-out comparisons for manipulation and driving tasks.

  • Complementary Supervision: Auxiliary perceptions (OCR, layout, GUI) are most valuable when used in combination with dense grounding; this full mixture performs best across both manipulation and driving scenarios.

3. Scaling and Composition Dependence:

The effectiveness of the model is contingent on both scale and supervision composition:

  • Scale: Scaling the grounding pretraining strengthens both the quality of the grounding itself and downstream action learning, with generalization tracking these perceptual gains most closely.

  • Composition: The combination of scale (for foundation quality) and complementary supervision (for effective action support) determines how well the model supports downstream actions.

**4. Distinct Foundations for Planning vs.

Improvements for AI systems

Based on the provided scientific paper, here are specific improvements for AI systems derived from GroundingPI, detailing what these improved systems can achieve:

)1. Enhanced Perceptual Foundation for System 1 Execution (Action-Oriented Models):

The core improvement is shifting the foundation model's focus from general visual knowledge to precise spatial grounding. By integrating GroundingPI, AI systems will gain the ability to reliably map language queries directly onto physical locations and structures in an image.

  1. Specific Capabilities of Improved Systems:

  • [Autonomous Driving]: The system can now perform high-precision, open-loop trajectory prediction by outputting six ordered waypoints with a consistent L2 metric error (averaging 0.296m on nuScenes), enabling safer and more accurate navigation planning than general VLMs which struggle with dense scenes and tiny objects.

  • [Robot Manipulation]: The system can execute fine-grained, dexterous manipulation tasks (on platforms like RoboTwin 2.0 or RoboCasa-GR1) with superior success rates (up to 66.20% on RoboTwin 2.0 Full), demonstrating robustness across distribution shifts (OOD settings) by grounding its understanding of objects and spatial relationships before attempting control.

  • [Data Efficiency]: The system can learn complex physical tasks from significantly less action data (e.g., achieving a high success rate at only 50% demonstration fractions on RoboCasa-GR1), reducing the reliance on massive, expensive labeled datasets for every new physical task.

  1. System Architecture and Training Improvements:

  • [Foundation Model Design]: Implement a Perception Before Action paradigm where the foundation model is explicitly trained to develop spatial intelligence (e.g., through relative depth cues or explicit coordinate prediction) before action adaptation begins, leading to more robust generalization out-of-distribution (OOD).

  • [Task-Conditioned Grounding]: Develop a unified grounding interface that supports multiple modalities—including dense object detection, referring expression grounding, OCR text extraction, and GUI element identification—all sharing the same quantized coordinate vocabulary. This allows for seamless switching between perception tasks within a single model structure.

  1. Data Strategy Improvements:

  • [Data Recipe Optimization]: Optimize data collection and annotation pipelines to prioritize Dense Grounding supervision, as this provides the strongest spatial core for downstream manipulation and driving performance.

  • [Catalytic Supervision]: Investigate using OCR supervision not just for transcription but as a catalyst that sharpens fine-grained visual features, thereby amplifying the benefits of dense spatial grounding on robot manipulation tasks.

  1. Model Selection and Deployment Strategy:

  • [Model Scaling Insight]: Utilize scaling studies to determine the optimal balance between pretraining scale (tokens) and data composition (grounding types) to maximize generalization, ensuring that larger models are effectively guided by precise spatial supervision rather than just general knowledge.

  • [System Specialization]: Recognize the distinction between broad reasoning models (System 2 for planning) and perception-native foundations (System 1 for execution), allowing engineers to select the appropriate foundation model based on whether the task demands high-level reasoning or reliable, low-latency physical interaction.

Abstract

Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. On RoboTwin 2.0, it outperforms every mainstream backbone we evaluate in all four out-of-distribution settings, by up to 24.8% relative to the strongest backbone. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. On nuScenes, used as the visual backbone, GroundingPI attains an average open-loop L2 error of 0.296 m. We systematically analyze GroundingPI's pretraining in scale and data composition. Downstream autonomous driving and robotic manipulation improve as the pretraining is scaled. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding's substantial benefits for both, and OCR's potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

Sources

Related papers