PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

PROMO introduces Preference-Conditioned Multi-Objective Reinforcement Learning (MORL) for quadrupedal robots, addressing the limitation of fixed scalar rewards by allowing operator intent to explicitly condition locomotion trade-offs at runtime. This framework matters because it enables a single policy to adapt its behavior dynamically based on deployment preferences—such as prioritizing tracking, stability, or efficiency—without requiring retraining or switching between specialized controllers.

The gist: A single policy is conditioned on preferences over three deployment-facing objectives—tracking, stability, and efficiency—while embodiment-specific locomotion priors remain fixed.

How it works

PROMO frames locomotion as a partially observed multiobjective decision process by separating semantic objectives from fixed locomotion priors. The core mechanism involves conditioning the policy on the preference vector, denoted as the runtime input to a single policy: A single policy is conditioned on preferences over three deployment-facing objectives—tracking, stability, and efficiency—while embodiment-specific locomotion priors remain fixed. This separation keeps embodiment-specific requirements internal to the controller and makes the preference vector a meaningful runtime interface rather than a reformulation of the full reward function.

The framework utilizes a semantic reward vector where the semantic objective index set is I = [track, stab, eff], with K = I = 3. The scalar semantic utility is defined as U(r sem t, w t) = w T t r sem t, where the preference vector defines the weights applied to these objectives. During training, this utility is augmented by a fixed locomotion prior term: During training, semantic utility is augmented by a fixed locomotion prior, u loc(w t) = w T t r sem t + λpriorr prior t, where λprior is a fixed coefficient.

Architecture and Factorization

PROMO extends the TeacherAligned Representations (TAR) architecture to incorporate preference conditioning through specialized encoders. The architecture features distinct latent representations: zP t = EP(s priv t, w t), zH t = EH(ot-H:t, w t), where both encoders are preference-conditioned. During training, the critic receives inputs including the preference-conditioned privileged latent zP t. The actor receives inputs that include the history latent and velocity estimator output: The actor receives (ot, zH t, v hat b t, w t) and outputs 12 joint-position targets.

A critical design choice is the Objective/prior factorization to address the mismatch between vector-valued MORL objectives and dense reward design. This involves: separating operator-facing semantic objectives from fixed embodiment-level locomotion priors. The fixed prior terms, such as contact timing and posture constraints, are represented by r prior t and are added to the actor's utility but do not define critic heads. This factorization is deemed critical for controllability, as the paper notes that the fixed-prior formulation attains a 6% fall rate under challenging evaluation conditions, whereas alternative designs can increase it significantly.

Optimization and Deployment

The training objective for preference wt is defined as: Jtrain(θ, w t) = w T t Jsem(πθ, w t) + λpriorJprior(πθ, w t), where λprior = 1 in all experiments. The actor optimization uses a scalarized reward: For actor optimization, the semantic reward is scalarized as r sem t(w) = w T t r sem t. Furthermore, the policy includes a small action-diversity regularizer to ensure preference responsiveness: The resulting actor objective is Lactor = LPPO + λdivLact div, which penalizes differences between policies conditioned on similar observations but different preferences.

Evaluation and Results

PROMO is evaluated across simulation and real-robot deployment using four main evaluation criteria: training performance, common semantic evaluation, preference-interface structure, and hardware deployment. In simulation, the dense 100-preference study shows that 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference–objective correlation of 0.843. On the Unitree Go2 hardware, changing only the preference produces significant changes: preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. The results establish that preference changes systematically modulate hardware behavior with a fixed policy, demonstrating that the interface is interpretable as it allows operators to specify why behavior should change—for example, by prioritizing stability over tracking—and the policy learns how to realize that trade-off.

Core Contributions

The paper outlines four core contributions.

Here are specific improvements to AI systems derived from the PROMO framework, along with what those improved systems can achieve:

)1. Implementation of a Semantic Preference Interface for Adaptive Control:

Instead of hard-coding fixed reward weights or relying on complex, opaque latent gait codes, the system will be conditioned on an explicit semantic preference vector (tracking, stability, efficiency). This allows for runtime switching of locomotion behavior by simply inputting a preference weight vector.

2. Decoupling Task Objectives from Embodiment Priors:

The architecture will maintain a fixed set of embodiment-specific locomotion priors (e.g., contact timing, posture constraints) that are optimized during training but remain constant during deployment. The semantic objectives (tracking, stability, efficiency) will be treated as the sole runtime control inputs. This prevents low-level shaping terms from becoming artificial objective dimensions and ensures that operator intent maps directly to high-level behavioral goals.

3. Robust Sim-to-Real Transfer via Preference Modulation:

The system can be deployed on physical hardware (e.g., Unitree Go2) without retraining or controller switching. The learned policy is inherently robust because it has been trained across a dense set of 100 preferences in simulation, and the hardware performance is shown to scale predictably based on the preference input alone (e.g., reducing specific energy by up to 30.4% when prioritizing efficiency).

4. Multi-Objective Specialization from a Single Policy:

The system can achieve objective specialization from one deployable policy. Rather than maintaining three separate controllers (one for tracking, one for stability, etc.), the single policy learns the ability to realize any desired trade-off on demand by modulating its internal objective weights.

5. Interpretable Control and Behavioral Intent:

The preference vector acts as an interpretable representation of behavioral intent. An operator can explicitly specify prioritize stability over tracking, and the policy is trained to realize that trade-off, rather than manipulating a latent gait code or reward function coefficients. This provides a transparent interface for mission planning and human oversight.

)What the Improved AI System Can Do:

The resulting AI system will be capable of performing complex quadrupedal tasks in dynamic environments where the required locomotion strategy shifts based on mission needs, terrain changes, or operator demands. Specifically:

1. Maximize Mission Flexibility: The robot can seamlessly transition between high-speed tracking (for following a fast target), stable maneuvering (for navigating rough terrain or recovering from disturbances), and energy-efficient cruising (for long-duration patrols) simply by changing the semantic preference input.

2. Achieve Optimized Performance on Demand: It can execute a best-of gait for any given situation—such as maximizing tracking accuracy while maintaining acceptable stability, or minimizing energy expenditure while ensuring the robot remains upright—without needing to be pre-trained for those specific conditions.

3. Ensure Robust Deployment: The robot can be deployed rapidly in the field, and its performance will be predictably modulated by the preference input, offering high reliability that surpasses fixed-objective controllers and specialized experts trained offline on only one objective.

4. Enable Semantic Command Execution: It moves beyond simple trajectory following to executing semantic commands (e.g., Be stable, Be efficient), allowing for higher-level task execution in autonomous agents where the desired outcome is defined by a trade-off, not a fixed trajectory.

Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective approach that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100 sampled preferences in simulation, 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference-objective correlation of 0.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at https://amrmousa.com/promo/.