From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator

summary

The gist

The gist Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries; this paper

In short

The research developed an efficient modular robotic system for autonomous shelf stocking in supermarkets. It integrates off-the-shelf hardware with ROS2 and uses Behavior Trees for planning, YOLO/SAM2 for vision, and Model Predictive Control (MPC) for precise navigation. Experiments showed over 98% success rate in pick-and-place tasks, though performance still lags behind human workers.

Key concepts

Behavior Trees (BTs): BTs are used to plan complex tasks by modeling high-level decisions reactively and fault-tolerantly. They consist of action nodes for specific movements and control nodes that manage the flow, using sequences to execute steps in order or selectors to try different paths until one succeeds.
Model Predictive Control (MPC): MPC is a control strategy used for 'last-meter' navigation. It solves an optimization problem in real-time to ensure safe and constrained motion by predicting future states. This allows the robot to precisely manage coupled base and head movements while respecting physical limits.
SAM2: SAM2 (Segment Anything Model 2) is a visual model used for advanced object segmentation and tracking. It helps the system accurately identify and isolate specific products in images or videos, which is crucial for the picking pipeline to determine exactly what needs to be manipulated.
Kalman Filter (KF): The Kalman Filter is employed to accurately locate the robot within cluttered environments like grocery aisles. It uses a predict-update cycle to fuse noisy sensor data, providing a more reliable estimate of the robot's precise position even when visual or lidar data is imperfect.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator · Read on arXiv

Davide Peron, Victor Nan Fernandez-Ayala, Lukas Segelmark

University of Padova · KTH Royal Institute of Technology

Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries. This paper introduces an efficient modular robotic system for autonomous shelf stocking, integrating commercially available hardware with a scalable algorithmic architecture. A major contribution of this work is the system integration of off-the-shelf hardware and ROS2-based perception, planning, and control into a single deployable platform for retail environments. Our solution leverages Behavior Trees (BTs) for task planning, fine-tuned vision models for object detection, and a two-step Model Predictive Control (MPC) framework for precise shelf navigation using ArUco markers. Laboratory experiments replicating realistic supermarket conditions demonstrate reliable performance, achieving over 98% success in pick-and-place operations across a total of more than 700 stocking events. However, our comparative benchmarks indicate that the performance and cost-effectiveness of current autonomous systems remain inferior to that of human workers, which we use to highlight key improvement areas and quantify the progress still required before widespread commercial deployment can realistically be achieved.

DOI: 10.1109/CASE69030.2026.11704237

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "From Pixels to Shelf".

Dev: The gist Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries;

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So, we're looking at the paper 'From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator' by Davide Peron, Victor Nan Fernandez-Ayala, and Lukas Segelmark. What does that title actually mean when you hear it?

Dev: It suggests we're talking about taking vision from pixels and turning it into shelves in a supermarket without a human doing all the work. It's about building a whole system that works together.

Rosa: Exactly. The focus here isn't just on one piece of hardware, but how you integrate off-the-shelf stuff with this kind of complex AI planning and control for real retail environments #pg1.

Dev: And the authors are showing how to combine ROS2 perception, planning, and control into one deployable platform for supermarkets #pg1.

Rosa: It sounds like they're tackling those huge problems of dynamic human interactions and tight spaces we talked about earlier in the discussion.

The paper's summary: Rosa: So, what does this paper actually summarize? Basically, they introduce this efficient modular robotic system designed for autonomous shelf stocking #pg1.

Dev: It uses Behavior Trees for task planning, fine-tuned vision models for object detection, and a two-step Model Predictive Control framework for navigation using ArUco markers #pg1.

Rosa: That sounds like a lot of machinery working together to figure out where to go and what to pick.

Dev: They use Behavior Trees, which are really good for modeling complex, highlevel decision processes because they're modular and fault-tolerant #pg2.

Rosa: And for the actual movement, they have this Model Predictive Control framework that handles the precise shelf navigation using those ArUco markers #pg1.

Dev: They also use a Kalman Filter to accurately locate the robot in cluttered areas where lidar can fail because of its minimum range limitation #pg3.

Rosa: So, it's a multi-layered approach, combining planning, perception, and control into one package for retail environments.

The paper's improvements: Dev: Now let's look at what the authors suggest they improved or added to this system in 'From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator'.

Rosa: They focused on integrating commercially available hardware with a scalable algorithmic architecture, which is a big part of their contribution #pg1.

Dev: Specifically, they integrated off-the-shelf hardware and ROS2-based perception, planning, and control into one single deployable platform for retail environments #pg1.

Rosa: They also used fine-tuned vision models for object detection and a two-step Model Predictive Control framework specifically for precise shelf navigation using ArUco markers #pg1.

Dev: They validated this in an overnight closed-door operation scenario under standard indoor lighting, which matches retailer practice to avoid customer congestion and open with restocked shelves #pg2.

Rosa: And they provided a quantitative outlook, including side-by-side benchmarks against teleoperation and human workers, a joint time-cost performance index, and a hardware/software gap decomposition #pg2.

Conclusion: Dev: So to wrap up the paper 'From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator', the main implication is that they have shown a way to achieve reliable performance in pick-and-place operations across more than seven hundred stocking events #pg1 <ref:2509.11740#pg1,From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking>.

Rosa: It shows that this integrated approach can work reliably in real supermarket conditions, achieving over ninety-eight percent success rate across all those operations #pg2 <ref:2509.11740#pg1>.

Dev: The paper also provides a hardware/software gap decomposition, which outlines concrete paths toward commercially viable retail robots #pg2.

Taro: From my side, what interests me is how they handle the real-world failures—they tested this in an overnight closed-door operation to match retailer practice #pg6.

Rosa: I agree with Taro. It’s not just about lab results; it's about whether it can handle the chaos of a real store environment #pg6.

Dev: And what I find interesting is that they compared it against teleoperation and human workers, showing that while autonomous stocking might be slower than those methods due to sequential task execution #pg7.

More episodes

← Home