Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip
cs.LG
Submitted: 2023-07-29
Updated: 2026-09-15
Comments: 23 pages, 15 figures, 5 tables. Substantially revised and extended from v2 with a new simulation-based differential-drive rover case study under hidden asymmetric wheel slip, additional transfer and robustness evaluations, revised framing, and an added coauthor. Previously titled "Dynamic Deep-Reinforcement-Learning Algorithm in Partially Observable Markov Decision Processes."
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) performs well in continuous-control tasks, but evolving hidden disturbances create partial observability: the agent must infer decision-relevant latent dynamics from
Terminology
Abstract
Reinforcement learning (RL) performs well in continuous-control tasks, but evolving hidden disturbances create partial observability: the agent must infer decision-relevant latent dynamics from interaction history. This study investigates how observation history, action history, history length, and network structure affect recurrent Twin Delayed Deep Deterministic Policy Gradient (TD3) agents. Three recurrent architectures are evaluated under controlled disturbances with different temporal characteristics. Results show that action history is particularly important when observed responses depend on previous actions, and that processing past and current action-observation information within a unified temporal sequence improves performance compared with using separate branches. We also introduce H-TD3, which reuses recurrent states generated by the actor to initialize the critic, reducing duplicated sequence processing. The architectures are further tested in a simulation-based differential-drive rover motion-regulation task under hidden asymmetric wheel slip. Recurrent architectures retain their advantage under the physically motivated multiplicative wheel-slip model, while policies trained with abstract temporally structured disturbances transfer more effectively to previously unseen wheel-slip dynamics than policies trained without disturbances. These findings provide practical guidance for recurrent RL under partial observability and evolving hidden disturbances.
Sources
- Partially Observable Markov Decision Processes (POMDPs) and Robotics
- Recurrent Reinforcement Learning: A Hybrid Approach
- On Improving Deep Reinforcement Learning for POMDPs
- Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
- Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation
- Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs
- Deep reinforcement learning oriented for real world dynamic scenarios
- On Separation Between Learning and Control in Partially Observed Markov Decision Processes
- Causal Reinforcement Learning using Observational and Interventional Data
- Memory-based control with recurrent neural networks
- Recurrent Off-policy Baselines for Memory-based Continuous Control
- Adaptive Guidance with Reinforcement Meta-Learning
- Continuous control with deep reinforcement learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks