STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models

Vision-language-action (VLA) models often lack interpretability and struggle to follow precise natural language instructions that encode spatial, temporal, and logical requirements. This paper proposes STEP, a hierarchical framework that uses Signal Temporal Logic (STL) as a shared representation connecting high-level language understanding with low-level robot execution.

The gist

STEP is a hybrid language-conditioned robot planning framework where System 2 decomposes natural language instructions into STL specifications, and System 1 executes these specifications using either learned policies or STL-guided Model Predictive Control (MPC), enabling precise constraint enforcement and runtime replanning.

How it works

The framework separates the process into a high-level System 2 module responsible for task interpretation and formalization, and a low-level System 1 module that executes actions through either learned policies or MPC. System 2 leverages a Vision-Language Model (VLM) to produce structured task specifications from language instructions, which are then compiled by a deterministic STL compiler into executable sequences of subtasks paired with STL formulas. This process involves several checks, such as verifying that the output matches the required JSON schema and ensuring each constraint is supported by the selected skill.

STL Integration and Execution Switching

Signal Temporal Logic (STL) serves as the formal interface between high-level reasoning and low-level execution, providing a compact language for encoding spatial, temporal, and logical constraints. System 1 dynamically selects its executor based on the subtask requirements: use learned policies for expressive skills or use STL-guided MPC when precise constraint satisfaction is required. During execution, an STL monitor evaluates the robustness of the active subtask formula. This robustness value provides a continuous measure of progress and constraint satisfaction, allowing System 1 to switch between modes, such as reassigning a subtask from a learned policy to MPC if monitoring detects a violation.

Monitoring and Replanning Mechanism

The framework incorporates several mechanisms for execution management. A History Log (Memory) is maintained during execution, storing the most recent records of observation, state, selected executor, STL robustness values, and events. When the STL monitor detects a violation or subtask completion, this log is passed to System 2 during a recall to provide context for updating the plan. This allows System 2 to reason over recent execution progress and failures without discarding the original task structure, enabling it to revise or repair the plan while remaining grounded in the original instruction.

System Components and Evaluation

The architecture is instantiated across four components: validate generated task plans, shape MPC objectives through robustness costs, monitor execution for constraint violations, and anchor replanning to the original instruction after failures. The system evaluates three research claims: Precise Language Following, Learned Policy/Motion Planning Switching, and Few-Shot Replanning. Evaluation on a real-world tabletop manipulation platform demonstrated that STL-guided planning achieves higher safe success rates across constraint types, showed that STL robustness provides a practical switching signal between execution modes, and that structured STL feedback enables more targeted and efficient recovery after intermediate failures compared to informal failure signals.

Real-World Implementation Details

The real-world implementation pipeline involves perception, language processing, and execution. Perception converts the workspace into a typed scene representation using a segmenter and VLM to recover world-frame poses for entities with typed roles. The language pipeline assembles structured context—a catalog of skills, current entity names with roles, and a ruleset—which is fed to the VLM to return an ordered list of subtasks, each carrying a skill name, parameter bindings, and subtask-local constraints. The System 1 executor then iterates these subtasks in order. For each step, it activates local safety constraints and runs a nominal MPC while the per-step STL monitor evaluates the compiled formula on the resulting trajectory. Future work suggests that learned policies are monitored by STL robustness but do not yet use the specification as an inference-time control signal.

Limitations

Current limitations include: only MPC-based skills directly optimize their actions with respect to the STL specification, and the high-level plan is represented as a sequence of subtasks, which limits the range of branching or cyclic behaviors that can be expressed. Furthermore, the system struggles in cluttered environments due to deliberately simple components like the MPC solver. The VLM can still produce incomplete or incorrect task specifications when fine spatial details are missing from the scene description.

References

[1] NVIDIA,:, J. Bjorck, F. Castaneda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y. Fang, D. Fox, ˜ F. Hu, S Huang., J Jang., Z Jiang., J Kautz., K Kundalia., L Lao., Z Li., Z Lin., K Lin., G Liu.

Here are the specific improvements that can be made to existing AI systems by implementing the framework proposed in this paper (STEP), and what those improved systems will be capable of doing:

The core improvement is shifting from learning a policy that happens to work to learning a policy that adheres rigorously to formal, verifiable specifications. This creates a bridge between the high-level, flexible reasoning of Vision-Language Models (VLMs) and the low-level, precise control required for physical tasks.

Here are the specific improvements and capabilities:

1. Maneuver from Coarse Intent to Formal Constraints:

2. Precise spatial, temporal, and logical requirements encoded in natural language instructions will be translated into a formal structure using Signal Temporal Logic (STL). This moves the AI beyond simple semantic understanding to explicit constraint representation.

3. Hierarchical Task Decomposition with Formal Grounding:

4. The VLM (System 2) will decompose complex, multi-step natural language instructions into a sequence of subtasks, where each subtask is explicitly accompanied by an STL specification defining its spatial boundaries, timing windows (e.g., within 5 to 8 seconds), and safety constraints (e.g., avoiding the stove). This prevents the VLM from generating vague plans that fail during execution due to unstated temporal or spatial requirements.

5. Hybrid Execution Strategy:

6. The system will dynamically switch between two execution modes at the subtask level:

7. Model-Predictive Control (MPC) for tasks requiring precise constraint satisfaction (e.g., place it exactly 8cm left), and learned policies for perceptually complex or contact-rich behaviors (e.g., grasping). This ensures safety and precision where needed, while maintaining the expressive power of learned skills elsewhere.

8. Runtime Constraint Monitoring and Deviation Detection:

9. During execution, an STL monitor will continuously evaluate the robustness of the active subtask formula against real-time sensory signals (e.g., current object poses). If robustness drops below a safety threshold (indicating a violation of a constraint), the system immediately triggers a formal recall to System 2 for replanning.

10. Targeted, Context-Aware Replanning:

11. When failures occur, the system will not simply regenerate the entire plan from scratch. Instead, it will use the structured failure feedback—which includes which specific STL constraint was violated and by how much (the robustness margin)—to generate a highly targeted revision of only the affected subtask parameters or sequence. This leads to significantly faster recovery than current methods that rely on informal error signals.

12. Increased Reliability in Complex Environments:

13. The resulting AI system will exhibit higher precision, reliability, and interpretability in real-world manipulation tasks (like stacking cubes or delicate placement) because the language intent is explicitly enforced by a formal mathematical structure rather than implicitly relied upon by a learned policy.

In summary, this framework transforms VLA agents from smart guessers into guaranteed executors of complex, constrained physical tasks.