MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles

arXiv:2506.00173 · cs.GR, cs.RO · Submitted 2025-05-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles".

Dev: MotionPersona introduces a novel real-time character controller that allows users to characterize their characters by specifying various attributes and projecting them into generated motions,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: To get into specifics about what this paper proposes, "MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles" essentially introduces a new controller that allows you to define a character through various attributes and then project those definitions directly into actual generated motions.

Dev: So the core idea is conditioning the motion prediction on multiple inputs at once—the desired movement direction, the specific body shape parameters from something like SMPL-X, and descriptive text about the character's personality or demographics.

Taro: What I find compelling is that they are trying to solve that fundamental problem where existing models can’t separate the mechanics of walking from *who* is walking; they’re aiming to disentangle motion content from character context.

Rosa: Exactly, and the paper points out that previous deep learning controllers struggled to do this because they couldn't distinguish between a happy elderly person's gait and just any gait, which limits their effectiveness for real-time control.

Dev: If we look at their methodology, they use an autoregressive motion diffusion model conditioned on those inputs to predict the clean motion from a noisy sample, which seems like a sophisticated way to handle the generation process.

Taro: I wonder how robust this conditioning is when you throw unexpected environmental disturbances at it; specifically, what happens when the world misbehaves and the character needs to react in an unpredictable way?

Rosa: That's where I'm curious about its real-world applicability; does this controller have a practical operational time frame before we run into issues outside of a clean lab environment?

Dev: We need to check their performance metrics on things like latency and failure modes, because if the loop rate dips too low, the whole system becomes unusable for any kind of responsive control.

The paper's summary: Rosa: Moving into the actual substance of "MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles," the paper summarizes their approach as presenting a single, unified model capable of animating characters with different specifications at the same time.

Dev: So they’ve combined several techniques to condition their motion diffusion model on those directional controls, the character's physique defined by SMPL-X vectors, and detailed text descriptions of traits like mental status and demographics.

Taro: What really stands out to me in the summary is how they use an encoder-only transformer to process all these different inputs as separate tokens before feeding them into a decoder that predicts the actual motion sequence.

Rosa: That means they are essentially treating each piece of character information—the direction, the body shape, and the text prompt—as distinct pieces of data that need to inform the final output motion.

Dev: And to make sure it looks physically sound, they incorporate several losses during training; specifically positional and velocity losses using forward kinematics based on those body parameters.

Taro: I’m also paying attention to their strategy for diversity, as they augment the SMPL-X body shape parameters with random perturbations to the vectors, which should help prevent the model from getting stuck in overly repetitive motion patterns.

Rosa: That perturbation technique is interesting because it directly addresses one of those limitations where models might generate motions that are too uniform, and it seems to be a key part of their attempt at character customization.

Dev: They also introduced an example-based characterization technique as a complementary conditioning mechanism, which means they can characterize the controller using just a small set of motion clips rather than needing massive amounts of training data for every new character.

The paper's improvements: Rosa: Now let's talk about the specific improvements they suggest in "MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles," which focus on enhancing how this system works beyond just the basic setup.

Dev: They introduce several enhancements to ensure physical plausibility, including a foot contact loss specifically designed to prevent artifacts during the training process, which is a smart move for locomotion tasks.

Taro: The introduction of Classifier-Free Guidance at runtime is significant because it gives the user direct control over how much influence past motion has on the generated future motion, which should be useful when we need fine-grained temporal adjustments.

Rosa: That guidance mechanism allows for a dynamic control loop where you can modulate the influence of past movement based on what you are trying to achieve in that specific moment.

Dev: Furthermore, they propose an in-diffusion blending technique to smooth out the transitions between the past and generated future motion by blending frames at each denoising step, which should help reduce temporal discontinuities or jittering.

Taro: I'm also interested in how they handle character customization; their method of augmenting body shape parameters with random noise, specifically = (beta one + eta, beta two:ten +), is a way to introduce variability without needing a completely new model for every single physical variation.

Rosa: That suggests the system is designed to be highly adaptable; it’s not just about one perfect character but about projecting characteristics onto a wide range of plausible movements.

Conclusion: Dev: Wrapping up our discussion on "MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles," the paper demonstrates a framework that successfully integrates physical shape parameters with textual character traits to generate real-time locomotion control.

Rosa: Essentially, it shows a method where you can define complex character specifications and then immediately project those definitions into high-quality motions while still responding to dynamic locomotion signals.

Taro: From my view, the ability to condition on multiple distinct inputs simultaneously—direction, body shape, and psychological state—is what makes this approach more useful than previous methods that struggled with disentangling these elements.

Dev: I'm concerned about the practical deployment regarding performance; we need to see how stable the loop rate stays under heavy load and what the latency profile looks like in a live system.

Rosa: I still want to know if this controller holds up when you take it out of the lab and into a dynamic, unpredictable environment for extended periods, or if its operational time is limited.

Taro: The implications for embodied AI are huge; imagine robots transitioning between emotional states while maintaining their physical structure based on these specifications; that's where the real autonomy potential lies.

Dev: And we should also consider how effectively the example-based characterization technique allows for quick adaptation to novel characters without extensive prior training data.

Rosa: So, "MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles" provides a solid foundation for creating truly versatile character animation systems that react to both external commands and internal personality settings.

The University of Hong Kong · Shandong University · *Adobe Research*

cs.GR, cs.RO

Submitted: 2025-05-30

Updated: 2026-10-01

Comments: 15 pages, 11 figures, webpage: https://motionpersona25.github.io/

Project page: https://motionpersona25.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: MotionPersona introduces a novel real-time character controller that allows users to characterize their characters by specifying various attributes and projecting them into generated motions,

Key concepts

SMPL-X vector
This is a mathematical representation that defines a character's 3D body shape and physique. It serves as the physical blueprint for the character, allowing the system to understand how different body types influence movement generation.
Autoregressive motion diffusion model
This is the core AI technique used to predict clean motion from noisy data. It works by iteratively refining a random sample of movement frames until it matches the desired clean motion, guided by various character inputs like desired trajectory and physical parameters.
Classifier-Free Guidance (CFG)
CFG is a method used during runtime to control how much influence past motion has on the generated future motion. It balances the influence of conditioned inputs against unconditioned samples to ensure the output adheres closely to the specified character traits.
Example-based characterization
This technique allows users to define a character's style or persona using only a small set of short motion clips, rather than needing extensive text descriptions. It provides an example-based conditioning mechanism for customizing the controller with minimal input.

Terminology

Summary

MotionPersona introduces a novel real-time character controller that allows users to characterize their characters by specifying various attributes and projecting them into generated motions, addressing limitations in existing deep learning-based controllers that produce homogeneous animations.

How it works

  1. The system is conditioned on directional control signals (including desired future root trajectory), the character’s physique (parameterized by the SMPL-X vector), and a detailed text describing character-specific traits such as demographics and mental status.

  2. It employs an autoregressive motion diffusion model, conditioned on these inputs, to predict clean motion from a noised sample: The model predicts the clean motion of future time frames:

  3. The process utilizes an encoder-only transformer to process multiple input conditions as separate tokens, which are then concatenated before being passed through a 4-layer transformer decoder to produce the predicted motion.

  4. To enhance diversity, body shape parameters are augmented by applying random perturbations to the SMPL-X vector: We augment the data by applying random perturbations to each vector β as follows: β˜ = (β1 + η, β2:10 + ũ), η ∼ N (0, 0.2), ũ ∼ N (0, 0.5).

  5. An example-based characterization technique is introduced as a complementary conditioning mechanism to enable controller customization via short motion clips: We develop an example-based characterization technique as complementary conditioning, enabling the controller to be characterized using only a small set of example motions.

Data and Characterization

  1. A comprehensive locomotion dataset was curated featuring 50 human subjects (26 male, 24 female) ranging in age from 5 to 68, capturing a wide range of physical and mental traits.

  2. Each participant performed locomotion in eight different physical mental, or emotional states (neutral, angry, happy, depressed, drunk, fearful, excited, and refreshed).

  3. Body shapes were fitted using Mosh++ [Mahmood et al. 2019] to the SMPL-X vector for each performer.

  4. Textual annotations were collected via questionnaires structured using a predefined template: Template 'A [age]-year-old [gender], who is [physical build, mental state, attitude, mindset, etc], and [some personal traits]. [He/She] is moving with [one of the states].

Training and Optimization

  1. The denoising objective enforces the predicted motion to be close to the ground truth clean sample: Lsamp = Et∼[1:T],x0∼q(x0 c) ŷ0 − x0 2.

  2. Geometric losses are applied to ensure physical plausibility, including positional and velocity losses using forward kinematics (FK): Lpos = ∥p(ŷ0, R (β)) − p(x0, R (β)) ∥2.

  3. A foot contact loss is introduced during training to avoid artifacts: Lfoot = ∑t ∈contact pos′foot(t, z)2 + vel′foot(t)2.

  4. Classifier-Free Guidance (CFG) is applied at runtime to control the influence of past motion: G (xt, t;cp, cft, β, ctx) = G (xt, t;cp = ∅, cft, β, ctx) + γ G (xt, t;cp, cft, β, ctx) − G (xt, t;cp = ∅, cft, β, ctx).

  5. In-diffusion blending is proposed to reduce discontinuities between past and generated future motion by blending the first few frames of the generated motion with the last frame of the past motion within each denoising step: x˜it = w(i) · c end p + (1 − w(i)) · xit, for i = 1, 2,..., M (4).

Evaluation and Generalization

  1. Quantitative evaluation metrics include Fréchet Pose Inception Distance (FPD), Diversity Score (Div.), Trajectory Positional/Directional Error (TPE/TDE), Foot Sliding Distance (FSD), Character Classification Accuracy (CCA), and R-Precision@3.

  2. The method was compared against baselines such as LMP, MANN+DeepPhase, and AMDM, consistently showing superior performance across most metrics: Ours LMP MANN+DP AMDM Ours LMP MM.

  3. Generalization to new characters was tested by instructing ChatGPT to create 600 new character specifications for unseen subjects.

  4. A user study involving 200 participants and Gemini-2.

Improvements for AI systems

Here are specific, actionable improvements to existing AI systems based on the principles and architecture of MotionPersona:

The core innovation of MotionPersona lies in unifying high-level character specifications (physical attributes via SMPL-X parameters and psychological traits via text prompts) with low-level dynamic control signals (desired future root trajectory) within a single, real-time, generative framework.

Here are the specific improvements and capabilities this system enables:

  1. A unified controller capable of generating high-quality character animations that dynamically adapt to both external physical controls (like joystick inputs) and internal/specified character traits (like age, emotional state).

  2. The ability to animate multiple characters simultaneously in a single scene, each with unique body shapes and personality traits.

  3. Character customization via few-shot learning: Allowing users to implant a new character into the model using only a small set of example motion clips (e.g., 10 seconds), bypassing the need for extensive manual parameter tuning or detailed text prompting.

  4. Robust generalization to unseen characters: The system can generate high-quality motions for characters whose specifications (shape and traits) were not present in the training data by leveraging learned correlations between shape and motion, combined with semantic knowledge from CLIP embeddings.

  5. Real-time performance: The use of a diffusion model architecture optimized for low-dimensional motion data (46x46 equivalent) allows for fast inference, enabling interactive control systems that respond dynamically to user inputs without significant latency.

  6. Improved motion fidelity through in-diffusion blending: By blending the past and predicted future motion within the denoising process, the system mitigates temporal discontinuities (jittering) between sequential frames, leading to smoother, more physically plausible locomotion.

  7. Enhanced character-specific alignment via CLIP conditioning: Utilizing a CLIP embedding of a detailed textual description allows the system to accurately align generated motion with complex, nuanced personality traits described in natural language.

This improved AI system can perform the following specific tasks:

  1. A video game engine could use it to implement highly realistic NPCs where an elderly character's shuffling gait (specified by text) can be instantly triggered and maintained, even when the player inputs a sudden sprint (dynamic control signal).

  2. In an embodied AI setting, a robot could exhibit behavior that changes based on its personality prompt—for example, transitioning from an angry state to a refreshed state while maintaining its physical body shape.

  3. A virtual avatar creation tool would allow users to generate entirely new characters by uploading just three short clips of desired movement and typing a few sentences describing the character's demeanor (e.g., a cheerful 60-year-old male).

  4. A training pipeline for motion synthesis could use this system to quickly adapt a base model to generate motions for novel body shapes (e.g., generating a motion for an anthropomorphic creature) simply by providing the new shape vector and text description, without requiring full re-training.

Abstract

We present MotionPersona, a generative framework for character-aware locomotion control, in which the motion for a command depends on the captured persona, the body shape, and the character's style. Unlike style, which one performer can vary at will, persona and body shape are coupled in capture: each performer is observed in only one body. The captured data therefore cannot uniquely determine which motion characteristics should follow the persona and which should change with the body, leaving unseen persona-body combinations unconstrained. We capture 48 performers, aged 5 to 68, under the same nine styles and seven commands, 44 of them with persona annotation. From this repeated-measures design, we identify two robust associations between body shape and gait. These measurements guide a cross-body specification of which characteristics should change and which should be preserved. We implement this specification through a physically informed retargeting pipeline, producing cross-body training data while penalizing penetration and foot skating. On this data we train a single generative controller. A shape-aware VAE compresses each motion block into a few latent tokens and renders them on a conditioned target body under explicit geometric supervision; over these tokens, a latent flow-matching prior generates persona- and style-conditioned motion in two sampling steps. The controller covers all captured personas, a wide family of SMPL-X target bodies, and nine styles in one model, and runs at 27 ms per block on two threads of a laptop CPU. We verify the framework at every stage, following the same gait descriptors from captured to retargeted to generated motion and sweeping each axis in isolation. To our knowledge, this is the first real-time locomotion controller that carries part of a captured persona's performer-specific variation across independently selected body shapes and styles.

Sources

Related papers