ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot

arXiv:2610.01258 · cs.RO · Submitted 2026-10-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot".

Rosa: Autonomous colonoscopic navigation remains challenging due to deformable anatomy, weak-texture and specular endoscopic visuals, and contact-rich viscoelastic interactions.

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So, we’ve discussed the title and the basic premise of this work, and now we’re digging into the core summary of 'ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot'.

Dev: Essentially, the paper lays out how they address the difficulties in colonoscopic navigation by integrating a multi-cue perception module with an Action Chunking Transformer policy.

Taro: The main idea is using RGB input augmented by relative depth and pseudo-elevation maps to form an observation called ot =

I rgb t, Dt, Et: , which explicitly encodes high-frequency topographic features like folds and fine relief.

Rosa: That explicit encoding is what makes a difference because it helps the policy see features it might otherwise miss due to weak texture or specular highlights in endoscopic visuals.

Dev: Then, this observation ot is fed into an Action Chunking Transformer policy which predicts a sequence of future actions over a lookahead horizon, rather than just one step.

Taro: This sequence prediction capability allows the system to plan multi-step paths and coordinate its movements more intelligently based on the long-term visual context.

Rosa: And to make sure those predicted actions are smooth, they use Temporal Ensembling to fuse multiple overlapping predictions into a final, kinematically continuous action command.

Dev: The paper concludes by demonstrating system-level validation on ex-vivo porcine colons, showing strong performance across straight, curved, and complex tortuous segments.

Taro: The results are compelling because they validate the entire pipeline from perception to control in a real anatomical setting under conditions that mimic real-world challenges well.

Rosa: So, in short, ColoACT is a system designed to navigate the inherent difficulties of colonoscopy by using enhanced geometric sensing and a transformer policy for sequential action prediction.

The paper's summary: Dev: Now that we’ve seen the summary, let’s talk about the specific improvements they propose in 'ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot'.

Rosa: What are the key technical enhancements they suggest to make this system better than previous attempts?

Dev: The first improvement involves the perception phase, which is augmenting the standard monocular RGB input with relative depth and that pseudo-elevation map calculated via Scharr gradients on normalized depth maps to create that explicit geometric encoding.

Taro: They argue that this specific way of encoding helps in feature extraction in weak-texture environments, making fold ridges much more distinguishable for the AI than just raw depth data does.

Rosa: Right, and then there’s the control enhancement where they move from predicting single steps to an Action Chunking Transformer policy that looks at a horizon of sixteen steps ahead.

Dev: That sequence prediction capability is designed to generate smoother control sequences, which addresses the issue of jitter inherent in single-step prediction methods.

Taro: And they follow that up with Temporal Ensembling to fuse these overlapping action chunks, which acts as a low-pass filter to ensure kinematic continuity and stability during movement.

Rosa: So it’s a layered approach: better input perception followed by sequential action planning, then temporal smoothing for execution refinement.

Dev: The paper also points out that the overall system-level validation on the BGER platform confirms these improvements lead to measurable performance gains in error metrics, showing reductions in ADE and FDE compared to simpler RGB-D baselines.

Taro: The finding from their ablation studies is that this explicit geometric encoding lets the model infer those high-frequency cues far ahead, which supports the idea that the perception enhancement is not just an add-on but a necessary component for long-horizon accuracy.

Rosa: It seems like they're pushing for a more holistic system where every part—from how it sees to how it plans and executes—is specifically tuned to handle the unique challenges of this application.

The paper's improvements: Dev: Wrapping up our discussion on 'ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot', we've covered the main points regarding its performance and design.

Rosa: So, to summarize, the main implication is that this system offers a way to tackle the navigation difficulties of deformable anatomy by explicitly encoding geometric information alongside sequential action chunking for smoother control.

Taro: The impact on autonomy is that it shows that using structured latent spaces to disentangle navigation features gives us confidence in building systems that can navigate complex, dynamic environments reliably.

Dev: From an engineering standpoint, the system’s success with the BGER platform proves that a well-designed control loop can achieve stable and continuous motion even with high latency issues if you manage them correctly through techniques like temporal ensembling.

Rosa: It really puts us in a good position to think about how we can apply these ideas to other areas where visual ambiguity and contact interactions are significant, as we wrap up our discussion on 'ColoACT'.

Taro: I just want to mention that future work on developing adaptive navigation for truly complex maneuvers, like triple bends, will be key to realizing the full potential of this system.

Rosa: So that’s a summary of what we’ve covered in today regarding the paper 'ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot'.

Dev: I think it’s been an interesting deep dive into how they've managed to integrate perception, planning, and execution to create a robust autonomous system.

Taro: It was definitely exciting to see how they used those geometric cues to improve the performance metrics on ex-vivo data.

Conclusion: Rosa: So we’ve seen how ColoACT uses enhanced geometric sensing and sequential action chunking to tackle the difficulties of autonomous navigation inside a colon, and now we’re wrapping up this segment with some final thoughts on its impact.

Dev: I think what they showed with the Action Chunking Transformer policy really addresses those latency concerns we always worry about in closed-loop control systems, even if it's a generative policy.

Taro: From my side, I’m really interested in how well this AI handles unexpected changes when the environment misbehaves, like dealing with those sharp curves we talked about.

Rosa: Exactly. The fact that they achieved success rates of over eighty-five percent in straight segments and seventy-two percent in curved ones on ex-vivo colons suggests a pretty solid foundation for real-world deployment outside the lab setting.

Dev: It’s those smoothness metrics that tell me a lot; using Temporal Ensembling to filter out that high-frequency jitter is critical for any physical robot operating in contact with tissue.

Taro: I agree, and what stands out is how they used the RGB-D-E input, specifically that elevation map derived from Scharr gradients, to give the policy extra visual context it needed.

Rosa: That explicit encoding really does seem to help the AI infer those high-frequency geometric cues far ahead, which is something we need when dealing with things as soft and deformable as internal organs.

Dev: It’s a good demonstration of how combining different types of sensory data—appearance, depth, and explicit topography—can significantly boost the reliability of an action chunking system.

Taro: The implication here is that for future autonomy in complex biological spaces, we need to move beyond just reacting to immediate obstacles and start encoding the larger topological structure of the environment.

Rosa: Right, so it’s not just about getting through a straight pipe; it’s about understanding the whole landscape.

Dev: And from an engineering standpoint, seeing those performance improvements after ablation studies gives us concrete data on exactly which components—like that pseudo-elevation map or the ensemble filtering—are providing the most value in terms of loop rate and stability.

Taro: I think it points toward a future where autonomous systems can navigate environments where perfect physical modeling isn't available, relying instead on learned geometric intuition.

Rosa: Well said. The paper "ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot" gives us a lot to think about regarding how we build more resilient robotic systems in physically challenging domains.

Jian Hu, Shujing He, Leixin Chang, Zongze Li, Ding Huang, Chaoyang Shi, Chengzhi Hu

Senior Member, IEEE

cs.RO

Submitted: 2026-10-01

Updated: 2026-10-01

Project page: https://adamhu1.github.io/ColoACT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: Autonomous colonoscopic navigation remains challenging due to deformable anatomy, weak-texture and specular endoscopic visuals, and contact-rich viscoelastic interactions.

Key concepts

RGB-D-E Input
The system uses three visual cues: standard RGB images (color), a depth map (distance), and an elevation map (derived from the depth map). The elevation map explicitly encodes fine surface details like folds and ridges, helping the robot distinguish complex geometric features in the colon's anatomy.
Action Chunking Transformer (ACT)
Instead of predicting one step at a time, this policy predicts a sequence of future actions (a 'chunk') over a 16-step horizon. This allows for smoother, more continuous control than single-step policies, which is crucial for navigating deformable and complex environments like the colon.
Temporal Ensembling
This technique acts as a low-pass filter on the policy's predictions. By synthesizing multiple overlapping action chunk predictions using an ensemble average, it smooths out high-frequency noise from the generative model, ensuring the robot executes kinematically continuous and safe movements.

Terminology

Summary

Autonomous colonoscopic navigation remains challenging due to deformable anatomy, weak-texture and specular endoscopic visuals, and contact-rich viscoelastic interactions. ColoACT proposes an autonomous navigation system that integrates an RGB-D-E based Action Chunking Transformer policy for smooth, continuous closed-loop control of a self-propelled Bevel-GearBased Endoscopic Robot (BGER).

The gist: The proposed RGB-D-E ACT navigation policy achieves success rates of 85.4% in straight segments and 72.5% in curved segments on ex-vivo porcine colons, demonstrating robust performance across complex tortuous anatomies.

How it works

  1. The system utilizes a multi-cue perception module that augments the monocular RGB input with a relative depth map (D) and a gradient-based pseudo-elevation map (E). This observation is denoted as ot = [I rgb t, Dt, Et], which combines appearance with explicit geometric constraints. The elevation map E is computed by applying the Scharr gradient operator to the normalized depth map D to explicitly encode high-frequency topographic features such as folds and fine-scale relief, making fold-ridge cues more distinguishable.

  2. The control mechanism employs an Action Chunking Transformer (ACT) policy that predicts a sequence of future actions rather than a single step. The observation ot is first encoded into a feature embedding evis via a modified ResNet-18 backbone fθ, resulting in evis ∈ R d (3). This visual observation is then processed by the CVAE encoder to approximate the posterior distribution qϕ(zot, at+k−1) conditioned on the current observation and ground-truth action chunk.

How it works

  1. The policy network functions as a decoder that generates an action chunk at time t+k−1 = at, where k=16 is the prediction horizon. This generation is conditioned on a latent style z sampled from a diagonal Gaussian distribution (4) and the visual context evis: aˆt+k−1 = πψ(z, evis, P) (5). The network is trained by minimizing the Evidence Lower Bound (ELBO), which balances reconstruction loss and KL-divergence loss.

  2. To ensure smooth control consistency and filter high-frequency jitter inherent in generative policies, Temporal Ensembling is employed. Since the policy predicts a horizon of k=16 steps, multiple overlapping predictions are synthesized using an ensemble average (Boxcar filter) to generate the final executed action at time t: aˆt = 1/Nt Σ i aˆ(t-i)t (7). This mechanism acts as a low-pass filter, effectively smoothing out high-frequency noise from the generative policy and ensuring kinematic continuity essential for endoscopic safety.

How it works

  1. The mechanical platform is a compact self-propelled BevelGearBased Endoscopic Robot (BGER), designed with dimensions of 24 × 20 × 34 mm. Traction is provided by biocompatible PDMS treads, and locomotion is powered by two independent coreless DC micro-motors with a reduction ratio of 700:1. A custom self-propelled Bevel-Gear mechanism vectorially cancels radial and tangential reaction forces, effectively isolating the motor shafts from bending loads and reducing mechanical jitter. The BGER perceives the environment through a distal monocular vision module supporting video capture up to 720p at 30 fps.

How it works

  1. The system's performance is validated through ex-vivo deployments on unseen porcine colons, covering straight/curved and tortuous anatomies. Ablations confirm the benefits of the proposed components: RGB-D-E input reduced ADE by 23.8% and FDE by 18.6% compared to the RGB-D baseline, proving that explicit geometric encoding allows the model to infer the colon’s high-frequency geometric cues far ahead. Furthermore, temporal ensembling suppressed high-frequency oscillation in angular jerk (Jω < 5 a.u.), confirming its role in ensuring smooth and continuous motion.

How it works

  1. Semantic analysis of the latent space confirms that the CVAE encoder successfully disentangles navigation features. The t-SNE visualization of the latent variable z shows a highly structured manifold organization within the latent space, with a clear gradient transitioning from negative to positive angular velocities, suggesting that the CVAE encoder has successfully disentangled the underlying geometric features of the navigation task. Grad-CAM visualizations further confirm that the policy demonstrates a sophisticated, target-driven pathfinding behavior, anchoring onto navigable lumen regions while maintaining awareness of adjacent topological boundaries.

How it works

  1. The system demonstrated strong adaptability in complex maneuvers.

Improvements for AI systems

Here are specific improvements that can be made to existing AI systems, based on the ColoACT framework described in the paper:

  1. Improve robustness against weak visual conditions (low texture/specular highlights) by integrating a multi-cue perception module that explicitly encodes high-frequency geometric information. This involves augmenting standard RGB input with:

  2. Inferred relative depth maps (using models like Depth Anything V2-Base).

  3. A gradient-based pseudo-elevation map, calculated via the Scharr gradient operator on normalized depth maps, to explicitly encode fold ridges and fine topographic relief that are often lost in pure depth or RGB data.

  4. Enhance control smoothness and temporal stability by replacing single-step action prediction with an Action Chunking Transformer (ACT) policy framework (based on CVAE). This system predicts a sequence of future actions over a lookahead horizon (e.g., 16 steps), allowing for the synthesis of smooth, continuous motor commands.

  5. Ensure kinematic continuity and eliminate high-frequency jitter by implementing Temporal Ensembling. This technique fuses multiple overlapping action chunk predictions, effectively acting as a low-pass filter to synthesize a final executed action that is kinematically stable and smooth, crucial for handling viscoelastic interactions.

  6. Improve long-horizon prediction fidelity and geometric intent alignment by incorporating the RGB-D-E input into the ACT policy. The explicit elevation map helps the policy distinguish between the lumen center (target) and colon wall (obstacle) more effectively, reducing Average Displacement Error (ADE) by inferring high-frequency cues far ahead.

  7. Develop adaptive navigation capabilities for complex, tortuous geometries (e.g., 90-degree turns or triple bends). The ACT policy's ability to encode long-horizon geometric context allows it to demonstrate multi-stage polarity reversal patterns, enabling the robot to execute continuous, oscillation-free switching between steering phases required for navigating sharp curves and bends.

  8. Improve interpretability and semantic understanding of policy behavior by utilizing Grad-CAM visualizations on the visual backbone. This can confirm that the policy is focusing on navigable lumen regions while simultaneously maintaining awareness of adjacent topological boundaries (e.g., haustral fold ridges), ensuring collision-free navigation based on meaningful geometric cues rather than simple obstacle avoidance.

  9. Mitigate the sim-to-real gap in learning-based policies by utilizing imitation learning from expert demonstrations, reducing the reliance on perfect physical modeling of complex viscoelastic tissue interactions during training.

Related papers