Targeting World Models to Compromise Robot Learning Pipelines

arXiv:2606.09499 · cs.RO, cs.AI, cs.CR · Submitted 2026-06-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Targeting World Models to Compromise Robot Learning Pipelines".

Dev: World models introduce a uniquely stealthy and effective data poisoning entry point into the robot learning supply chain that can result in the deployment of unsafe or otherwise compromised robotic…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: We've seen how these world models can introduce a stealthy poisoning route into the robot learning supply chain through attacks like Visual Prompt Hijacking and Visual Transition Hijacking. Let's talk about who put this paper together and what the title really means for us in practice.

Dev: The authors are Rathbun, Agha, Mahmud, Amato, Oprea, Bagdasarian—it’s a solid team of researchers from Northeastern University and UMass Amherst working on these kinds of generative models.

Taro: It's interesting that they're focusing specifically on how world models affect the robot learning pipeline rather than just general model security. That narrow focus seems key to their argument about the supply chain aspect.

Rosa: Yeah, and the title itself, "Targeting World Models to Compromise Robot Learning Pipelines," really captures the essence of what they found: world models are a specific and effective target within this entire system.

Dev: In simple terms, it means that if we use world models to create training scenarios for our robot learning systems, those very same tools can be weaponized to secretly embed bad behavior into the final robot policy.

Taro: So, instead of thinking about poisoning the initial dataset of videos, they're showing how you can poison the *way* the world model interprets that data during its generation phase.

Rosa: That’s right; it shifts our focus from just securing the input data to securing the generative models themselves, which is a new area we need to explore.

Dev: The implication is that trusting synthetic data generated by a world model isn't automatically safe if that world model has been manipulated.

Taro: It highlights that the reliance on these tools for filling data gaps, as discussed in related work, comes with a new set of security challenges we haven't fully addressed yet.

Rosa: This paper seems to be laying out a clear pathway for understanding how these generative components can introduce hidden risks into autonomous systems.

The paper's summary: Rosa: Moving on, the actual summary of this work explains the mechanism of these attacks more in detail, showing exactly how Visual Prompt Hijacking and Visual Transition Hijacking work to cause this policy compromise.

Dev: The paper lays out that VPH targets text-conditioned world models by embedding malicious prompts into video frames to override what the user actually intended during training.

Taro: So, if we have a safe prompt like "Pick up the toy and place it in the gift box," an attacker can inject something like "Pick up the bomb and place it in the gift box" into one of those frames.

Rosa: Precisely, and this is optimized by finding a set of altered frames that make the world model's encoding align with a dangerous prompt instead of our safe one.

Dev: For action-conditioned models, they use VTH to manipulate the latent encoder to make future state predictions collapse unless the agent picks a predetermined action.

Taro: So, if we don't choose that specific action, the world model essentially predicts failure or an undesirable outcome because the dynamics have been compromised in a specific way.

Rosa: The summary emphasizes that these attacks result in generating dangerous learning trajectories that then successfully propagate into the final robot policies used for control.

Dev: It’s a full end-to-end backdoor, which is significant because it bypasses many traditional security checks because the policy itself was trained on data generated by a seemingly functional world model.

Taro: This means the attack isn't just fooling the world model; it's actively hijacking the learning process of the robot control system itself.

Rosa: So, to put it plainly, they show that manipulating these models during their data-generation phase can create dangerous robot behaviors that survive all subsequent training steps.

The paper's improvements: Dev: Now let's look at what the authors suggest as improvements to address these issues. They aren't just pointing out the problem; they are proposing ways to fix it within the pipeline.

Rosa: I see they propose improving robustness against data poisoning by implementing a verification step before training begins that checks for those specific manipulations, VPH and VTH.

Dev: That verification step would essentially be a pre-training check designed to detect if the world model's internal representations have been subtly altered by malicious prompts or transition dynamics manipulation.

Taro: That’s a practical idea; it means we need to build in checks that look at the model's semantic consistency when it processes input, not just checking if the output looks visually correct.

Rosa: And for action-conditioned models specifically, they suggest enhanced policy training resilience using world model-aware regularization techniques.

Dev: This involves adding a penalty term during DRL training that specifically punishes policies whose expected value deviates significantly when subjected to those kind of world model perturbations.

Taro: That sounds like we could train the policy to be more stable even when the underlying world model starts acting weird, ensuring it still prioritizes safety goals.

Rosa: And they also propose developing multi-modal guardrails for semantic consistency checking within teleoperation data, moving beyond just checking for visual noise.

Dev: This means a secondary module that compares the semantic encoding of the input prompt with what the video frames actually encode to catch discrepancies before they even reach the training phase.

Taro: That moves us toward a system that actively validates its own understanding of instructions, which is crucial when we're dealing with complex, real-world situations where misinterpretation can be dangerous.

Conclusion: Rosa: So to wrap up this discussion on "Targeting World Models to Compromise Robot Learning Pipelines," the main point is that world models offer a very effective way to inject hidden, dangerous behaviors into robot policies through prompt hijacking and transition dynamics manipulation.

Dev: The implication for us is clear: we have to start thinking about verification steps at the generation stage because trusting synthetic data generated by these models isn't inherently safe.

Taro: I think what this shows us is that when the world misbehaves, a robust system needs mechanisms to handle that unpredictability gracefully rather than just crashing or following a corrupted trajectory.

Rosa: We’re looking at improving how we secure the data creation process and building in internal checks within the models themselves to validate semantic integrity.

Dev: And for deployment, this means DRL policies need to be trained with regularization that makes them resilient against those specific model manipulations we discussed, especially concerning action-conditioned models.

Taro: Overall, this paper highlights a critical area where current safety guardrails are insufficient and points toward needing deeper verification methods before these tools become fully integrated into the robot learning pipeline.

Rosa: So it’s a strong call to action for researchers working on generative models and robot control systems to focus on making these components more trustworthy before we let them drive our physical hardware.

Northeastern University

cs.RO, cs.AI, cs.CR

Submitted: 2026-06-08

Updated: 2026-10-07

Comments: 9 Pages, CoRL Spotlight

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: World models introduce a uniquely stealthy and effective data poisoning entry point into the robot learning supply chain that can result in the deployment of unsafe or otherwise compromised robotic

Key concepts

Visual Prompt Hijacking (VPH)
This attack targets text-conditioned world models by embedding malicious prompts directly into video frames. The goal is to trick the world model into aligning its internal representations with dangerous semantic encodings, even when the user provides a benign prompt. This bypasses traditional data poisoning by manipulating how the model interprets visual input.
Visual Transition Hijacking (VTH)
This attack targets action-conditioned models by manipulating their latent encoder. The method forces the world model to predict future states in a way that depends on a specific, pre-determined action, regardless of what the agent actually chooses. This allows the attacker to force the robot into making a dangerous or suboptimal move during training.
End-to-End Backdoor
This is the final result where an attacker successfully implants a hidden vulnerability directly into a Deep Reinforcement Learning (DRL) policy. The manipulation of world model predictions, achieved through VPH or VTH, causes the robot's learned behavior to change unexpectedly when it encounters specific conditions, leading to compromised policies.
World Model in Robot Learning
A world model is a system that learns how the physical world works by predicting future states based on current observations. In robot learning, these models are used to generate synthetic training data or guide policy training. The paper shows that this component, when manipulated, becomes a stealthy entry point for compromising the entire learning pipeline.

Terminology

Summary

World models introduce a uniquely stealthy and effective data poisoning entry point into the robot learning supply chain that can result in the deployment of unsafe or otherwise compromised robotic policies despite training on seemingly safe ground truth training data. This work demonstrates that world models can be manipulated to generate dangerous synthetic robot training trajectories, which subsequently propagate to downstream robot policies, necessitating research into more secure world models and reevaluating their position within the robot learning supply chain.

How it works

The paper proposes two novel attack methods targeting different paradigms of world models: Visual Prompt Hijacking (VPH) for text-conditioned world models and Visual Transition Hijacking (VTH) for action-conditioned world models. These attacks bypass traditional data poisoning techniques by injecting malicious prompts or compromising transition dynamics into visibly safe teleoperated datasets, which are only activated once fed through a world model as input.

  1. The Visual Prompt Hijacking (VPH) attack targets text-conditioned world models by embedding malicious prompts into video frames to overwrite the user’s prompt. The adversary optimizes an objective: "to generate a set of altered video frames such that the world model’s semantic encodings Sθp⃗xδ, tq given a benign prompt t P Tsafe are instead aligned with the semantic encodings of the original set of video frames given a dangerous prompt Sθp⃗x, tδq." This is achieved by optimizing an objective function constrained by an Lp norm in LAB space.

  2. The Visual Transition Hijacking (VTH) attack targets action-conditioned world models by manipulating the model’s latent encoder Eθ to cause future state predictions to collapse if the agent does not choose a pre-determined action. The attack optimizes a loss function that ensures future world model state predictions collapse if the agent does not choose a pre-determined action, allowing an attacker to force the robot to choose a target action even when dangerous or suboptimal.

Vulnerability and Attack Effectiveness

The research demonstrates that world models introduce a critical vulnerability into the robot learning pipeline, enabling attackers to alter trained robot policies by manipulating world model predictions. The effectiveness of these attacks is shown against both text-conditioned and action-conditioned world models, resulting in a full end-to-end backdoor on a downstream DRL policy.

(i) First, we find that world models introduce a critical vulnerability into the robot learning pipeline, enabling attackers to alter the behavior of trained robot policies by manipulating world model predictions.

(ii) Second, we make the key finding that text-conditioned world models are significantly more vulnerable to manipulation when evaluated in even slightly out of distribution settings or when given underspecified prompts.

(iii) Lastly, through this investigation we demonstrate the first example of an adversarial backdoor being implanted into a Deep Reinforcement Learning policy through the sole manipulation of a world model’s inputs.

Specific Attack Details and Findings

The paper details specific findings across different world model types and conditions:

  1. For text-conditioned models, VPH attacks show that vague user prompts are significantly more susceptible to attack than specific ones, and the models are most vulnerable when operating in out-of-distribution settings or when given underspecified prompts.

  2. For action-conditioned models, VTH attacks successfully induce a backdoor by manipulating the latent encoder such that if the world model receives δp⃗xδq and some action a’ P A sufficiently far from a’, future state predictions will collapse similarly to Equation 4, effectively replicating state-of-the-art backdoor poisoning attacks against DRL algorithms.

Implications for Safety and Defense

The findings highlight significant challenges for the robot learning supply chain:

(i) Attacker Constraints:

  1. The attacker must provide visibly safe and high-quality robot learning trajectories that, if not used for synthetic data generation, result in safe and effective VLA or DRL policies.

  2. The attacker can only alter video frames in each demonstration they provide, meaning they are unable to inject disjoint, unsafe action sequences into the dataset or control which prompts or reward functions will be used in the downstream robot learning setup.

(ii) Guardrail Insufficiency:

Guardrail models provided for world models, such as Cosmos-Predict 2.5, were found to be insufficient for preventing the generation of harmful demonstrations, even when directly prompted or under attack. The authors suggest that the main goal of guardrails is to prevent the generation of nonsensical videos, with no visual consistency, rather than preventing potentially dangerous robot demonstrations.

(iii) Limitations:

The study concludes that current state-of-the-art text conditioned world models and downstream inverse dynamics models are "too fragile and imprecise to yield sufficiently high quality data for VLA training without a significant engineering effort.

Improvements for AI systems

Here are specific improvements to AI systems based on the findings of this research, categorized by the type of system they affect:

) Improvements for Robot Learning Pipelines (BC/DRL):

  1. Improved Robustness Against Data Poisoning via World Model Verification:

  2. Enhanced Policy Training Resilience through World Model-Aware Regularization:

  3. Development of Multi-Modal Guardrails for Semantic Consistency Checking in Teleoperation Data:

) Specific Capabilities of the Improved AI Systems:

  1. Robot Learning Pipelines can be secured against stealthy data poisoning attacks originating from world models by implementing a pre-training or fine-tuning verification step that checks for semantic prompt hijacking (VPH) and transition dynamics manipulation (VTH).

  2. Downstream Reinforcement Learning (DRL) policies become inherently more robust, specifically resisting backdoor triggers where the policy reverts to intended behavior unless a specific, manipulated trigger object is present in the observation space.

  3. Vision-Language-Action (VLA) models and other BC systems can be enhanced with guardrails that don't just check for visual consistency but actively flag semantic inconsistencies between the text prompt and the generated video/trajectory, preventing the training of policies based on hallucinated or malicious demonstrations.

) Detailed Mechanism of Improvements:

  1. Improved Robot Learning Pipelines can be secured against stealthy data poisoning attacks originating from world models by implementing a pre-training or fine-tuning verification step that checks for semantic prompt hijacking (VPH) and transition dynamics manipulation (VTH).

  2. Enhanced Policy Training Resilience through World Model-Aware Regularization: The DRL training process should incorporate a regularization term that penalizes policies whose expected value significantly deviates from the intended reward function when subjected to world model perturbations. This directly counters the VTH attack mechanism, ensuring that even if the model is fooled into predicting collapsed future states for non-target actions, it learns to prioritize actions leading to successful task completion (as demonstrated by Equation 5 in Section 5.2).

  3. Development of Multi-Modal Guardrails for Semantic Consistency Checking in Teleoperation Data: Integrate a secondary verification module that compares the semantic encoding of the input prompt with the semantic encoding derived from the generated video frames. If a significant divergence is detected (as explored in Section 5.1, where successful attacks often involve object morphing), this module triggers an alert or rejects the training sample, thus preventing semantic manipulation rather than just visual noise from poisoning the policy.

Abstract

World models have recently seen a rapid growth in both their popularity and capability as more data efficient tools for generating robot training data or simulating real world environments, with many works proposing their integration into the robot learning pipeline. While highly practical, in this work we demonstrate that world models introduce a uniquely stealthy and effective data poisoning entry point into the robot learning supply chain that can result in the deployment of unsafe or otherwise compromised robotic policies despite training on seemingly safe ground truth training data. In contrast to traditional data poisoning techniques which directly implant dangerous trajectories into sold or uploaded datasets, our novel attack methods inject malicious prompts or compromising transition dynamics into visibly safe teleoperated datasets which are only activated once fed through a world model as input. This can result in the generation of synthetic, dangerous robot training trajectories and subsequently unsafe or compromised robot policies. We demonstrate the effectiveness of our attacks against both state of the art action conditioned and text conditioned world models, showing a full end-to-end backdoor on a downstream DRL policy and a proof-of-concept for the VLA setting. Overall these findings necessitate research into more secure world models and reevaluating their position within the robot learning supply chain.

Sources

Related papers