Generalizable Dense Reward for Long-Horizon Robotic Tasks

arXiv:2604.00055 · cs.RO, cs.CV, cs.LG · Submitted 2026-03-31 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Generalizable Dense Reward for Long-Horizon Robotic Tasks".

Dev: Existing robotic foundation policies trained primarily via large-scale imitation learning often struggle with long-horizon tasks due to distribution shift and error accumulation,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're talking about a paper called "Generalizable Dense Reward for Long-Horizon Robotic Tasks," and I’m curious what that title actually means in practice for us here in the field robotic community.

Dev: I’ve seen the title, and it sounds like they are tackling one of the big headaches with current foundation policies, which is how they handle tasks that take a long time to complete without any immediate feedback.

Taro: It suggests they’re moving away from just relying on imitation learning because that method struggles when things get complex or unexpected during a long sequence of actions.

Rosa: Exactly, it points toward creating a reward system that doesn't require perfect task-specific instruction for every single step along the way.

Dev: I think the authors are proposing a way to give the AI both high-level direction and local guidance simultaneously, which is something we’ve struggled with in our own loop rate constraints.

The paper's summary: Rosa: To summarize what this paper proposes, it’s basically this new reward framework called VLLR that combines two main things: an extrinsic reward coming from large language models and vision-language models for tracking task progress, and a second part that is an intrinsic reward based on how certain the policy is about its own actions.

Dev: That makes sense; so instead of just a single score at the end of a long mission, they’re giving the agent constant updates on whether it’s making progress toward the overall goal.

Taro: I see that as solving the credit assignment problem for long sequences by breaking down what needs to be done into smaller, verifiable subtasks using those LLMs.

Rosa: Right, and then they use VLMs to look at the visual scene and figure out which subgoal is currently being worked on, giving a progress estimate between zero and one.

Dev: That estimation then gets turned into a reward signal based on the change in that progress estimate from one step to the next.

The paper's improvements: Rosa: One of the main points they highlight is how this VLLR framework is structured in two stages to be computationally efficient, which helps avoid running expensive VLM queries constantly during training.

Dev: That two-stage approach sounds like a smart way to manage those computational costs; using the VLM signal only for initializing the value function for two hundred thousand steps before switching over is quite strategic.

Taro: I’m interested in that second stage, where they use policy self-certainty as the intrinsic reward during fine-tuning with PPO and sparse task success rewards.

Rosa: That self-certainty metric quantifies the concentration of the action distribution, which they interpret as a way to check if the policy's internal representation of how things work is consistent and moving in the right direction.

Dev: So, while Stage I uses semantic supervision for structure, Stage II relies on that intrinsic signal for dense feedback at every step without needing those heavy models running constantly.

Conclusion: Rosa: So to wrap up what we’ve heard about this paper, VLLR offers a structured way to use semantic understanding from LLMs and VLMs alongside internal policy certainty to build rewards for long-horizon tasks that are much more robust than previous methods.

Dev: I think the core idea here is that you can get coarse supervision through task decomposition and fine-grained guidance through the intrinsic self-certainty reward during the policy refinement phase.

Taro: It seems like this structured distillation of semantic information combined with modeling internal progress provides a viable path for extending foundation model fine-tuning into real robotic control.

Rosa: And that's where I think we should focus next—how does this system actually perform when it’s pushed outside the controlled lab environment, and can we expect it to maintain that level of performance over very long operational periods?

Dev: That’s the practical question, Rosa; if we can keep those inference costs down during training and ensure the loop rate doesn't suffer under real-world latency, then this framework could really move us closer to deploying more capable autonomous systems.

Carnegie Mellon University

cs.RO, cs.CV, cs.LG

Submitted: 2026-03-31

Updated: 2026-10-05

Comments: Accepted at IROS 2026. Project page: https://silongyong.github.io/vllr_project_page/

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: Existing robotic foundation policies trained primarily via large-scale imitation learning often struggle with long-horizon tasks due to distribution shift and error accumulation, necessitating a

Key concepts

Task Decomposition
A Large Language Model (LLM) takes a high-level instruction and the scene's structure to create an ordered sequence of achievable subgoals. This breaks down a complex goal into manageable steps, ensuring the agent plans coherently by progressively reducing uncertainty about what needs to be done next.
VLM Progress Estimation
A Vision-Language Model (VLM) evaluates the agent's current visual state against the planned subgoals to estimate how much progress has been made toward the overall task. This estimation is used to generate a reward signal that encourages movement toward achieving those specific intermediate goals.
Policy Self-Certainty (RSC)
This intrinsic reward measures how concentrated the policy's action distribution is at each step. It quantifies the internal consistency of the policy's latent world model, acting as a dense feedback signal to ensure the agent is taking confident and coherent actions.
Two-Stage Optimization
The framework uses a two-stage training process. Stage I initializes the value function using extrinsic rewards for 200K steps to learn task structure. Stage II fine-tunes the policy using intrinsic self-certainty and sparse task rewards to refine local control, ensuring both global planning and step-by-step execution are optimized.

Terminology

Summary

Existing robotic foundation policies trained primarily via large-scale imitation learning often struggle with long-horizon tasks due to distribution shift and error accumulation, necessitating a novel dense reward framework that combines extrinsic semantic supervision with intrinsic policy certainty.

The gist

VLLR is a dense reward framework combining (1) an extrinsic reward from Large Language Models (LLMs) and Vision-Language Models (VLMs) for task progress recognition, and (2) an intrinsic reward based on policy self-certainty.

How it works: Subgoal Decomposition for Extrinsic Reward

The framework addresses the need for semantic supervision at the level of subgoals to alleviate sparse credit assignment problems. This is achieved through a four-step design for the extrinsic reward:

  1. Task Decomposition with Large Language Models: A LLM maps a high-level task instruction and a structured scene graph into an ordered sequence of grounded subgoals, denoted as mapping LLM: (G, I) → Gk K k=1. This decomposition is conditioned on global structural information to generate a coherent plan that progressively reduces spatial and semantic uncertainty.

  2. Progress Estimation with Vision-Language Models: A VLM evaluates the agent’s visual observations against the ordered subgoals to determine which subgoal is currently being attempted and estimate the resulting progress toward the overall task, yielding a scalar progress estimate pt ∈ [0, 1]. This estimation is then transformed into a reward using the formula RV LM(st, at, st+1) = ˆpt+1 − pˆt.

  3. Correcting Raw Progress Estimation: The raw VLM progress estimation is corrected by stabilizing the estimates before computing the running maximum progress variable pˆt+1 = max(ˆpt, pt+1), which tracks the highest observed progress estimate to ensure rewards are only assigned when new progress is achieved.

  4. VLM Reward as Proxy for Value Function Initialization: To maintain computational efficiency, the VLM-derived progress signal is used solely during a short initialization phase (200K steps) to supervise the value function, avoiding prohibitive inference costs during full training.

How it works: Intrinsic Reward via Policy Self-Certainty

To satisfy the second requirement—providing dense per-step feedback to guide local action refinement—the framework introduces policy self-certainty as an intrinsic reward signal. This metric quantifies the concentration of the action distribution, serving as a proxy for whether the policy’s latent world model is internally consistent and progressing toward task completion.

The self-certainty (RSC) is formally defined as: RSC = − 1 / A Σ A i=1 log(A · πθ(i)), where A denotes the size of the discrete action space and πθ(i) is the probability assigned by the policy to action i. This intrinsic reward is provided in a per-step manner, acting as a dense signal complementary to the coarse-grained extrinsic reward.

How it works: Two-Stage Optimization Procedure

The final reward model, Rfull, integrates these components under a two-stage optimization framework: Rfull = α 1[Stage I] RV LM + β 1[Stage II] RSC + ϕ 1[Stage II] Rtask.

** Stage I (Value Initialization): The extrinsic reward (RV LM) is used solely to initialize the value function for 200K steps. No further VLM queries are performed afterward. In this stage, α = 1.**

** Stage II (Policy Finetuning): The policy is fine-tuned using PPO with the intrinsic self-certainty reward (RSC) and sparse task success rewards (Rtask), without further VLM queries. In this stage, β = 0.1 and ϕ = 10.**

This hierarchical design ensures that subgoal supervision provides coarse-level structure, intrinsic self-certainty provides fine-grained action-level guidance, and the sparse task success reward anchors the optimization towards the true task goal.

Key Findings and Contributions

The evaluation on the CHORES benchmark demonstrated that VLLR achieves up to 56% absolute success rate gains over the pretrained policy on in-distribution tasks, consistently outperforming state-of-the-art RL finetuning methods based solely on sparse rewards. Furthermore, VLLR generalizes to out-of-distribution task compositions, achieving up to 10% higher success rates than prior methods. Ablation studies revealed that self-certainty primarily enhances success rates, particularly on out-of-distribution tasks, while VLM-based value initialization improves task completion efficiency by encoding high-level task structure during the initial phase. This demonstrates that structured semantic distillation combined with intrinsic progress modeling provides a practical pathway for extending foundation model fine-tuning to long-horizon robotic control.

Improvements for AI systems

Here are the specific improvements that can be made to existing AI systems by implementing the VLLR framework, and what those improved systems will be capable of:

The implementation of VisionLanguage-Long-horizon Reward (VLLR) offers three primary, synergistic improvements over current robotic foundation policies trained via imitation learning or sparse RL:

  2. Inference Cost Reduction during Training via VLM Initialization.

  4. Improved Task Success Rate and Generalization on Complex, Unseen Scenarios (Out-of-Distribution Tasks).

Specifically, the improved AI systems will be capable of the following:

  2. The system will significantly reduce the computational cost of training long-horizon policies by using a Vision Language Model (VLM) only during a brief initialization phase to set up an informed value function, rather than querying it at every single environment step throughout the entire fine-tuning process. This allows for much faster and more scalable training cycles without incurring prohibitive inference latency.

  4. The system will achieve substantially higher success rates (up to 10% gains on out-of-distribution tasks) compared to state-of-the-art RL finetuning methods that rely solely on sparse rewards or manual reward engineering. This is because the VLLR framework provides a dense, hierarchical reward signal derived from semantic task progress (LLMs/VLMs) and intrinsic policy self-certainty, which guides the policy toward correct subgoals and internally consistent world models.

The resulting improved AI systems will possess these specific capabilities:

  2. They can be fine-tuned much more efficiently because they only require expensive VLM queries during the initial value function setup, leading to faster convergence in terms of total training steps while maintaining high performance levels.

  4. They will exhibit superior robustness when faced with novel or out-of-distribution tasks (e.g., new object affordances or relational reasoning) because the learned reward structure is grounded in semantic task hierarchy (LLM decomposition) rather than brittle, task-specific manual shaping rewards, allowing them to generalize their skills effectively to unseen environments.

Sources

Related papers