BAT: Balancing Agility and Stability via Online Policy Switching for Long-Horizon Whole-Body Humanoid Control

Developing a unified framework that can achieve agile, precise, and robust whole-body behaviors—particularly in long-horizon tasks—remains challenging due to conflicting control requirements such as stable manipulation versus highly dynamic responses. This work proposes BAT, an online policy-switching framework that dynamically selects between two complementary whole-body Reinforcement Learning controllers to balance agility and stability across different motion contexts.

The gist

BAT is an online policy-switching framework that orchestrates complementary control policies to leverage their respective strengths for long-horizon, multi-task scenarios by dynamically selecting the most suitable policy based on the current motion context.

Framework Components and Motivation

The paper addresses the trade-off between decoupled whole-body policies, which provide stable and disturbance-robust behaviors, and coupled whole-body policies, which enable agile and dynamic motions but often sacrifice precision. The framework is motivated by the need for a unified approach that leverages both paradigms for long-horizon tasks requiring both highly dynamic motions (e.g., jumping over obstacles) and precise, stable manipulation (e.g., standing manipulation with minimal disturbance). To solve the challenge of online policy switching, the authors leverage hierarchical reinforcement learning (HRL) guided by sliding-horizon option evaluation to mitigate the absence of ground-truth signals and address temporal credit assignment issues where switching benefits are delayed.

Key Mechanisms for Policy Selection

BAT integrates three main modules to facilitate dynamic decision-making:

1. An Option-Guided Hierarchical Reinforcement Learning (HRL) framework that selects a discrete controller index, choosing between the decoupled policy (πD) and the coupled policy (πC), based on an observation constructed from proprioception, motion reference, and previous state. The high-level switching policy is optimized using a composite reward function that includes velocity tracking rewards, pose tracking rewards, regularization terms for smoothness and energy efficiency, and a soft alignment term to guide switching behavior.

2. An Option-Aware Vector-Quantized Variational Autoencoder (VQ-VAE) that encodes motions into discrete tokens. This VQ-VAE is trained with an objective that aligns the token space with switching decisions by introducing an option prediction head, encouraging latent tokens to separate motions favoring different controllers, thereby capturing controller-dependent structure relevant for switching.

3. A Decision Fusion Module that selects the final controller by trusting the high-level policy when in-distribution and deferring to the option prediction head when outside its training support. This module estimates distributional shift using three signals: H(πsw), DKL(Pcb∥Ptrain), and H(pϕ), fusing them to determine the final selection, ensuring reliability even when local value differences are high.

Data Construction and Guidance

The framework relies on extensive offline data construction for structured guidance. This involves two primary steps:

1. Sliding-Horizon Policy Pre-Evaluation: This step assesses switching decisions locally in time by evaluating each option (πD or πC) over a finite window of length H, isolating local motion segments and preventing short-lived instability from dominating long-horizon performance. The expert decision is defined as the one maximizing the option return within that window.

2. Offline Data Collection: This step applies sliding-horizon evaluation over retargeted motion data from both policies (πD and πC), generating high-quality switching demonstrations data (DOp) via motion blending with inertialization to ensure smooth transitions at clip boundaries.

Experimental Validation

Extensive simulations and real-world experiments validate BAT across diverse scenarios. In simulation, the option-aware VQ-VAE representation was shown to achieve higher reward and success rates compared to raw motion representations, indicating that improved representation quality enhances policy identification. When tested on sequential multi-motion tasks, BAT achieved the best overall performance by learning improved switching strategies through exploration, demonstrating the effective integration of HRL and option guidance. Hardware deployment on the Unitree G1 humanoid confirmed that BAT enables successful execution of sequential tasks by dynamically switching between policies to handle diverse motion phases within a unified framework, consistently selecting the more suitable policy based on motion characteristics.

Conclusion

BAT successfully balances agility and stability for long-horizon whole-body control by orchestrating complementary controllers through an option-guided HRL framework and an option-aware VQ-VAE. The approach demonstrates that combining structured guidance with HRL provides a more effective solution than purely reward-based switching, leading to improved performance across single-motion and sequential multi-motion tasks in both simulation and hardware. BAT proves versatile by dynamically selecting the appropriate policy based on the motion context, achieving superior success rates over competing baselines. The framework is modular and validated for real-world deployment on humanoid platforms.

How it works

The framework is structured around an Option-Guided Hierarchical Reinforcement Learning (HRL) setup where a high-level switching policy selects between two frozen low-level controllers, πD (decoupled) and πC (coupled).

Here are the specific improvements and capabilities enabled by implementing the BAT framework, based on the provided paper:

) Robust Long-Horizon Whole-Body Control with Adaptive Agility/Stability Trade-off:

The improved system (BAT) can execute complex, multi-stage long-horizon tasks—such as navigating rough terrain while simultaneously performing precise manipulation—by dynamically switching between two complementary whole-body control paradigms.

1. The system can switch to the decoupled policy for stable, disturbance-robust behaviors (e.g., standing on uneven ground or fine object placement).

2. The system can switch to the coupled policy for highly dynamic, agile motions (e.g., jumping over obstacles or rapid locomotion).

) Enhanced Sample Efficiency via Option-Guided Hierarchical Reinforcement Learning (HRL):

By using offline option guidance derived from sliding-horizon policy pre-evaluation, the system drastically reduces the required number of interaction samples needed to learn effective switching policies.

1. The system learns optimal switching patterns directly from expert demonstrations (DOp) without relying solely on sparse, delayed reward signals.

2. This leads to significantly faster training convergence and better credit assignment for long-horizon decisions compared to standard HRL or value-function based switching methods, reducing the overall computational cost of policy learning by up to a factor related to the sample complexity reduction (Equation 6).

) Superior Motion Representation via Option-Aware VQ-VAE:

The system utilizes an option-aware VQ-VAE that explicitly learns latent representations tailored to distinguish between motion phases favored by different controllers.

1. The learned discrete tokens capture controller-dependent structural features in the motion space, which the decision fusion module uses to make more informed switching choices.

2. This results in higher accuracy in identifying task contexts, as evidenced by superior performance metrics (Table I and Fig 4), allowing the system to select the correct policy based on subtle differences in motion structure rather than just raw proprioception.

) Reliable Online Switching via Confidence-Weighted Fusion:

The final decision-making module employs a confidence-weighted fusion strategy, integrating three distinct signals: the high-level switching policy's output, the predicted option preference from the VQ-VAE, and a distributional shift measure (DKL divergence).

1. When uncertainty is low (in-distribution), it trusts the learned switching policy.

2. When uncertainty is high or the motion context is ambiguous, it falls back to the option prediction head, providing a reliable oracle estimate without requiring perfect transition exposure during training.

) Improved Task Performance and Generalization:

The integrated framework enables superior performance across diverse tasks compared to using either the decoupled or coupled policies in isolation.

1. In simulation, BAT achieves higher success rates and lower tracking errors than both the pure decoupled and pure coupled baselines (Table III).

2. On real hardware (Unitree G1), the system demonstrates versatility, successfully executing sequential long-horizon tasks that require a continuous interplay of stable manipulation, locomotion, and dynamic maneuvers.