BiPO: Bidirectional Partial Occlusion Network for Text-to-Motion Synthesis

Generating natural and expressive human motions from textual descriptions is challenging due to the complexity of coordinating full-body dynamics and capturing nuanced motion patterns over extended sequences that accurately reflect the given text. BiPO introduces a novel model that enhances text-to-motion synthesis by integrating part-based generation with a bidirectional autoregressive architecture, allowing it to consider both past and future contexts while enhancing detailed control over individual body parts without requiring ground-truth motion length.

The gist

BiPO is the first model integrating part-based generation with bidirectional autoregressive architecture for text-to-motion synthesis without requiring ground-truth motion length, achieving state-of-the-art performance on HumanML3D by combining detailed control over individual body parts with global motion coherence.

Model Architecture and Training Objectives

BiPO is built upon a transformer-based architecture that integrates both unidirectional and bidirectional processing when employing part-based motion generation. The model utilizes 6 lightweight VQ-VAEs to discretize part motions, where the Root part has a reduced dimension of 64, while other parts have a token dimension of 256. A Selective Part Coordination Layer is added before all remaining layers except the first transformer layer, where each layer within this layer shares its weights and includes three MLP layers. For text-to-motion generation, the CLIP model with the ViT-B/32 variant is used to encode text features.

The training objective combines both unidirectional and bidirectional masking schemes to reconstruct each part’s motion sequence conditioned on the text embedding. The loss function for each body part is defined as:

L i hybrid = −Ec∼p(c) [λXL l=1 log pθ(c i lMuc) + (1 − λ)XL l=1 log pθ(c i lMbp)], where λ controls the balance between unidirectional and bidirectional masking, with optimal performance achieved when λ = 0.5.

Part-based Generation and Masking Strategies

The model introduces a part-based bidirectional autoregressive approach where each body part is independently modeled while leveraging bidirectional context from other parts to maintain overall motion coherence. To control the flow of information and prevent excessive interdependency among body parts, two types of attention masks are employed during training:

1. Unidirectional Causal Mask (Muc): This enforces autoregressive generation, where only the text token c i 0 is unmasked initially, and all motion tokens c i 1:L and the [END] token c i L+1 are masked.

2. Bidirectional Part-based Mask (Mbp): This leverages bidirectional context by unmasking both the text token c0 and the [END] token c i L+1, along with a randomly selected subset of motion tokens c i 1:L for each part. The mask Mbp is defined to control attention based on unmasked tokens U = [u0, u1, u2,...].

Partial Occlusion (PO) Technique

To relax the dependency among body parts and preserve the independence of each part’s representation, BiPO devises the Partial Occlusion (PO) technique. PO is a stochastic training technique that utilizes uniform random masking to selectively occlude motion tokens from specific body parts during training. Instead of consistently providing full motion data, PO randomly occludes certain tokens, supplying only partial information about other body parts. This encourages each part’s motion generator to learn to produce coherent motions even when information from other parts is incomplete. The coordination of the current part motion c i coord can then be computed as c i coord = LN(c i + MLPi(ˆc j)), where ˆc j denotes the motion tokens, including those masked tokens by PO.

Inference Process

During inference, BiPO generates the final motion sequence in a two-stage process:

1. Initial unidirectional generation phase: The model generates each body part’s motion sequentially, conditioning on its own past tokens and the current tokens of all other parts.

2. Bidirectional refinement step: A bidirectional autoregressive approach is applied to selectively mask and regenerate tokens, specifically those at even indexed time steps in the previously generated sequence, utilizing both past and future contexts to resolve inconsistencies and fine-tune the motions for better alignment with overall temporal dynamics.

Performance Evaluation

BiPO achieves state-of-the-art performance on HumanML3D, outperforming recent methods such as ParCo, MoMask, and BAMM in terms of FID scores and overall motion quality. In comparative evaluations on the KIT-ML test set, BiPO demonstrates strong generalization capability. The model excels not only in text-to-motion generation but also in motion editing tasks (Temporal Inpainting, Temporal Outpainting, Prefix, and Suffix), where it outperforms existing methods in FID scores.

Here are specific, actionable improvements for AI systems based on the BiPO (Bidirectional Partial Occlusion Network) paper, along with what these improved systems can achieve:

1. Mediocre Text-to-Motion Systems will be replaced by a system capable of generating highly nuanced, physically plausible human actions with precise control over specific body parts.

2. The system will be able to perform complex, long-duration motion synthesis (e.g., entire sentences or paragraphs) where previously models failed due to lack of temporal coherence and inter-part coordination (e.g., walk forward carefully with arms extended resulting in naturally symmetrical transitions).

3. The system will enable high-fidelity motion editing, allowing users to precisely modify specific segments of a generated motion (e.g., Temporal Inpainting or Prefix generation) while maintaining global consistency across the entire body, rather than producing jarring discontinuities.

4. The system will achieve state-of-the-art realism and diversity on benchmark datasets like HumanML3D, outperforming existing models (ParCo, MoMask, BAMM) in metrics crucial for visual quality (FID) and semantic alignment (R-Precision).

5. The system will possess superior interpretability concerning text prompts; it can accurately differentiate between fine-grained actions involving specific limbs or body segments (e.g., never leaves the ground, hops) rather than producing generic, poorly coordinated movements.

6. The system will be robust to incomplete input data, allowing it to maintain high coordination and realistic synthesis even when partial motion information from other body parts is missing during training or inference (due to the Partial Occlusion technique).

7. The system will function effectively in multimodal scenarios where text descriptions must be accurately mapped onto 3D skeletal representations, ensuring that the generated motion features are semantically close to their textual counterparts (low MM-Dist).

8. The system can generate diverse variations of a single action from the same text prompt (high MModality), enabling content creators or virtual agents to explore varied stylistic interpretations of an action.

These improvements translate directly into AI systems that are:

1. A high-end tool for the film, game, and VR industries requiring complex character animation driven by natural language.

2. A sophisticated interactive agent capable of executing complex, multi-step physical commands based on conversational input (e.g., a virtual assistant that can execute nuanced physical gestures).

3. A robust content creation pipeline where artists can rapidly prototype and iterate on character movements with high fidelity, significantly reducing the need for extensive manual motion capture or keyframe animation.

Generating natural and expressive human motions from textual descriptions is challenging due to the complexity of coordinating full-body dynamics and capturing nuanced motion patterns over extended sequences that accurately reflect the given text. To address this, we introduce BiPO, Bidirectional Partial Occlusion Network for Text-to-Motion Synthesis, a novel model that enhances text-to-motion synthesis by integrating part-based generation with a bidirectional autoregressive architecture. This integration allows BiPO to consider both past and future contexts during generation while enhancing detailed control over individual body parts without requiring ground-truth motion length. To relax the interdependency among body parts caused by the integration, we devise the Partial Occlusion technique, which probabilistically occludes the certain motion part information during training. In our comprehensive experiments, BiPO achieves state-of-the-art performance on the HumanML3D dataset, outperforming recent methods such as ParCo, MoMask, and BAMM in terms of FID scores and overall motion quality. Notably, BiPO excels not only in the text-to-motion generation task but also in motion editing tasks that synthesize motion based on partially generated motion sequences and textual descriptions. These results reveal the BiPO's effectiveness in advancing text-to-motion synthesis and its potential for practical applications.