Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation

arXiv:2509.18597 · cs.RO · Submitted 2025-09-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation".

Dev: Large language models (LLMs)-based code generation for robotic manipulation has recently shown promise by directly translating human instructions into executable code, but existing approaches are limited by language ambiguity,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So, to recap where we are, we've touched on how this paper proposes a human-in-the-loop framework that uses lifelong skill learning and code generation to handle ultra-long manipulation tasks. Essentially, the thesis is that existing LLM approaches struggle with the ambiguity and context window limitations inherent in complex robotic planning because they can’t effectively learn and adapt skills incrementally through human feedback.

Dev: That’s right, and what they claim is that by encoding user feedback directly into reusable skills and extending their functionality over time via a user-designed curriculum, the framework becomes much more robust for these long tasks. They show this approach leads to a zero point nine three success rate and a forty-two percent efficiency improvement in feedback rounds when solving extremely long-horizon tasks, like building a house that requires planning over twenty primitives.

Taro: I think the key takeaway from their summary is that they are tackling the problem of task decomposition by integrating human verification into the learning loop, which is different from just letting an LLM try to map a whole task at once.

Rosa: Exactly, and they structure this process into three main phases: first, preference-aligned skill acquisition where users clarify what skill to learn through multi-turn interaction; second, lifelong capability extension where the agent expands functionality for unseen cases using a user curriculum; and finally, task-specific retrieval and planning which uses RAG to pull in relevant skills from external memory.

Dev: The method relies on several key components we need to keep in mind: they use external memory with Retrieval-Augmented Generation, indexing examples by instructions and skills by their docstrings, and they employ a hint mechanism to guide the agent when retrieval is insufficient.

Taro: I'm interested in the "hint mechanism" because it addresses a real limitation of RAG systems—the risk of retrieving irrelevant data—by giving the user an explicit way to steer the system toward the right skill when things get ambiguous.

Rosa: It sounds like they are trying to balance the power between automation and human oversight, ensuring that while the AI is learning continuously, it remains firmly under human direction throughout every phase of development.

Dev: The entire structure is designed specifically to preserve previous functionalities while enabling dynamic in-context adaptation through that continuous human guidance, which tackles the stability issues common in long-running code generation processes.

Taro: So, the paper is proposing a system that doesn't just generate a plan; it generates and refines a library of skills over time, making it more resilient to the unpredictable nature of real-world manipulation environments.

Rosa: That resilience seems key, especially when we consider deploying these systems outside of perfectly controlled lab settings where things rarely go exactly as expected.

Conclusion: Rosa: Looking at the full picture of this work, the title "Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation" really captures the essence of what they achieved: moving beyond just generating a single block of code for a task to creating an evolving skill set that is guided by human input.

Dev: I think what it means in simpler terms is that we are building systems where the robot doesn't just follow instructions; it learns and develops its own robust toolkit incrementally, with us acting as the continuous teacher and curator of that knowledge.

Taro: The real implication for autonomy is that this suggests a path toward much more reliable long-horizon behavior because instead of relying on massive pre-training or perfect initial planning, we can use iterative human correction to fine-tune the system's capabilities over many tasks.

Rosa: That iteration seems vital because when things go wrong in physical manipulation, the ability to pause, correct based on specific feedback, and then have that correction permanently stored as a skill is incredibly valuable for deployment.

Dev: From an engineering standpoint, it implies that the latency and loop rate management needs to be tight because this framework involves constant interaction between the agent generating code and us providing hints or feedback during the learning phases.

Taro: I agree with Dev on that; if we want this out in reality, we need to ensure the external memory retrieval and skill application happen fast enough that the human guidance doesn't get stuck waiting for a slow response.

Rosa: So, while this paper provides a solid methodology for building these skills, the future work will likely involve testing how well these learned skills hold up when deployed on truly heterogeneous robot arms and in open-world environments where they haven't seen those specific examples before.

Dev: That’s the next logical step; proving that the curriculum extension works reliably when faced with completely novel physical challenges, not just variations of known tasks.

Taro: Ultimately, this research points toward a future where complex robotic manipulation tasks are handled not by one monolithic AI solution but by a collection of specialized, continuously improving skills managed collaboratively between human and machine.

School of Computation, Information and Technology, Technical University of Munich

cs.RO

Submitted: 2025-09-23

Updated: 2026-10-07

Comments: final submission RA-L 2026.09

Code: https://github.com/google-research/google-research

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Large language models (LLMs)-based code generation for robotic manipulation has recently shown promise by directly translating human instructions into executable code, but existing approaches are

Key concepts

Preference-Aligned Skill Acquisition
This phase involves users guiding the agent to learn a specific skill by testing various requirements within the same scenario. The interaction ensures the agent learns behaviors that align with human preferences, making the initial skill acquisition more accurate and tailored to desired outcomes.
Lifelong Learning-Oriented Capability Extension
To handle new situations, this phase expands existing skills using a user-designed curriculum. It prevents forgetting old knowledge by either creating new named skills or extending current ones with modular code, ensuring the robot can perform novel, unseen tasks over time.
Retrieval-Augmented Generation (RAG)
The system uses vector databases to store learned skills and examples. When facing a new instruction, it retrieves the most relevant past examples and skill descriptions to include in the prompt. This allows the agent to access historical knowledge for planning long sequences of actions.
Hint Mechanism
A simple user-guided mechanism lets humans intervene when the retrieval process is insufficient or yields irrelevant data. Users can suggest specific previously learned skills, directing the agent to pause and teach a missing sub-behavior, improving task planning.

Terminology

Summary

Large language models (LLMs)-based code generation for robotic manipulation has recently shown promise by directly translating human instructions into executable code, but existing approaches are limited by language ambiguity, noisy outputs, and limited context windows, which makes long-horizon tasks hard to solve. This paper proposes a human-in-the-loop lifelong skill learning and code generation framework that encodes feedback into reusable skills and extends their functionality over time to robustly handle extremely long-horizon manipulation tasks.

The gist

Our framework achieves a 0.93 success rate (up to 27% higher than baselines) and a 42% efficiency improvement in feedback rounds when solving extremely long-horizon tasks such as “build a house,” which requires planning over 20 primitives.

How it works

The framework is structured into three phases: preference-aligned skill acquisition, lifelong capability extension, and task-specific retrieval and planning. This pipeline is designed to preserve previous functionalities while enabling dynamic in-context adaptation through human guidance.

  1. Preference-Aligned Skill Acquisition: This phase focuses on learning a new skill by clarifying what skill to learn, the initial task environment, and the expected functionality or behavior through multi-turn interaction with the user. The agent is initiated with a fixed set of skills and examples that demonstrate their use. Users work together to learn the skill by testing multiple requirements within the same scenario, allowing for preference alignment.

  2. Lifelong Learning-Oriented Capability Extension: To handle variations beyond a single task's distribution, this phase introduces a user-designed curriculum that enables bottom-up skill functionality expansion. The agent must solve more complex user-defined task variations and preserve and extend skill functionality to unseen cases while storing skills permanently in external memory. This prevents catastrophic forgetting by either creating new named skills or extending existing ones using modularized code like if-else or match-case.

  3. Task-Specific Retrieval and Planning: For long-horizon planning, the framework employs Retrieval-Augmented Generation (RAG) to retrieve relevant skills and examples from an external memory. This is supported by a simple yet effective hint mechanism that allows users to guide the agent when retrieval alone is insufficient, ensuring only the most relevant skills are applied. The system uses two vector databases: one for few-shot examples indexed by instructions and another for skills indexed by their docstrings.

Key Components and Mechanisms

The framework relies on several key components to manage knowledge and planning:

(1) External Memory with Retrieval-Augmented Generation (RAG):

An external memory stores learned skills and examples to prevent catastrophic forgetting. Retrieval is performed using vector databases (ChromaDB) indexed by task instructions for few-shot examples and by skill docstrings for skills. For a new instruction, the system retrieves the K=10 most similar examples based on cosine similarity and relevant skill headers from memory and appends them to the prompt.

(2) Hint Mechanism:

A simple yet effective hint mechanism allows users to guide the agent when retrieval fails or when irrelevant data is retrieved. Hints enable users to specify which previously learned skill may help with the current task, which is especially useful during skill learning and task planning, signaling where the agent needs to pause and teach a missing sub-behaviour.

(3) Dynamic Prompting:

The framework utilizes dynamic prompting for in-context learning. It employs specialized prompts such as the main actor prompt for flat code generation, the skill learning prompt which requires implementing a specific function while trying to preserve the previous functionality, and the task setup prompts for environment configuration.

Experimental Validation

The effectiveness of LYRA is validated across simulation benchmarks (Ravens, Franka Kitchen, MetaWorld) and real-world settings using a Franka FR3. Empirical analysis shows that skill-oriented human-in-the-loop feedback is more efficient than task-specific flat code feedback from LLMs. The framework achieves a high success rate of 0.93 and demonstrates a 42% efficiency improvement in correction rounds compared with LLM-based closed-loop methods. Notably, the framework can robustly solve extremely long-horizon tasks such as “build a house” requiring over 20 primitives in both simulation and reality.

Real-World Deployment

The learned skills are embodiment-agnostic and can be deployed on heterogeneous robot arms. The agent is deployed on a Franka FR3 using ROS2, leveraging state-of-the-art perception foundation models like Grounded SAM 2 for open-world object pose estimation and FoundationPose for orientation estimation in the real world. This allows the framework to robustly perform challenging long-horizon tasks in reality, including building a house, stacking a Jenga tower, and writing “ICLR.” The framework is further validated on generalization to unseen tasks such as “construct a temple” in real-world settings.

Improvements for AI systems

As a fastidious researcher, I have thoroughly reviewed the proposed framework, LYRA (Lifelong learning code synthesis framework with human-in-the-loop for Robotic long-horizon skill Acquisition). The core innovation lies in decoupling task planning from monolithic LLM reasoning by encoding feedback into reusable skills stored in an external memory and retrieved via RAG, all guided by a hint mechanism.

Here are the specific, high-impact improvements this framework enables for AI systems:

) Improvements to AI Systems Enabled by LYRA Framework:

  1. [] Improve generalization across unseen long-horizon tasks (e.g., build a house generalization).

  2. [] Enhance robustness against catastrophic forgetting during iterative refinement (human-in-the-loop updates).

  3. [] Increase feedback efficiency and reduce the number of required correction rounds in complex robotic tasks.

) Specific System Capabilities Enabled:

  1. [] The improved system can perform build a house (requiring 20+ primitives) reliably in both simulation and real-world settings, achieving a 0.93 success rate, significantly outperforming baselines like DAHLIA and CaP on novel structures (e.g., building a 4x3x3 pyramid).

  2. [] The system can learn and execute highly precise, preference-aligned skills (e.g., stacking blocks corner-to-corner with exact spacing) by incorporating human feedback, leading to more reliable physical execution than LLM-only closed-loop methods.

  3. [] The agent can dynamically adapt its code plan in real-time during interaction by retrieving the most relevant pre-learned skills and examples from an external memory (RAG), preventing prompt overload and hallucination, as evidenced by a 42% efficiency improvement in correction rounds compared to LLM-based closed-loop methods.

  4. [] The system can solve complex, multi-step robotic sequences (like build a temple) in the real world by reusing foundational skills (e.g., build jenga layer) learned from simpler tasks, demonstrating emergent capability through bottom-up skill expansion guided by user curriculum design.

  5. [] The system exhibits superior efficiency when correcting low-level execution errors; the hint mechanism allows users to direct retrieval to specific known behaviors, drastically reducing unnecessary corrections compared to relying on general LLM feedback or random prompt sampling.

Abstract

Large language models (LLMs) can translate natural-language instructions for robotic manipulation into executable code, but ambiguity, noisy generations, and limited context windows make ultra-long-horizon tasks unreliable. Closed-loop approaches that rely only on LLM feedback also struggle because LLMs have limited robotic reasoning, even when task errors are obvious to humans. Feedback is often stored in representations that generalize poorly to unseen tasks and can cause catastrophic forgetting as new corrections accumulate. We propose LYRA, a human-guided lifelong skill learning and code generation framework that distills human feedback into modular, reusable skills and incrementally extends their functionality across successive interactions while preserving previously learned behavior. External memory stores learned skills and execution examples; retrieval-augmented generation selects relevant knowledge, while user hints guide reuse when retrieval is insufficient, supporting ultra-long-horizon execution. Experiments on Ravens, Franka Kitchen, LIBERO-long, MetaWorld, and real-world tasks show a 0.93 success rate, up to 27% higher than baselines, and a 42% improvement in correction efficiency. LYRA also robustly solves ``build a house'', which requires planning over 20 primitives.

Sources

Related papers