E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning

E2HiL introduces a novel framework designed to significantly enhance the efficiency of real-world Human-in-the-Loop (HIL) Reinforcement Learning. The paper addresses the critical bottleneck in deploying advanced robotic systems: the substantial time, cost, and risk associated with collecting sufficient, high-quality data through physical interaction with human experts. By integrating entropy maximization into the sample selection process, E2HiL ensures that training efforts are focused on the most informative and uncertain states, thereby achieving efficient real-world policy learning while maintaining safety constraints imposed by human intervention.

The Challenge of Sample Efficiency in HIL RL

Traditional reinforcement learning methods often suffer from poor sample efficiency, a problem exacerbated when integrating human feedback. When humans guide a robot or provide expert demonstrations, the resulting dataset can be sparse or biased toward common trajectories, leading to policies that fail catastrophically when encountering novel situations. The core challenge is how to optimally utilize limited human time and expertise. E2HiL posits that simply collecting more data is not sufficient; rather, the data must be maximally informative. The paper argues that the key to efficient HIL learning lies in intelligently identifying state-action pairs where the current policy's uncertainty is highest or where expert disagreement is most pronounced.

Entropy Guidance for Sample Selection

At its heart, E2HiL leverages entropy as a principled measure of both uncertainty and potential information gain. The framework modifies standard sample collection protocols by introducing an entropy-guided objective function that dictates which samples should be prioritized for human review or interaction. This mechanism ensures that the policy is actively steered away from regions of the state space where it already performs adequately, instead focusing on the boundaries of current knowledge. Specifically, E2HiL implements a selection strategy based on:

1. Policy Uncertainty: Prioritizing states where the learned Q-values or action distributions exhibit high variance, indicating potential failure modes.

2. Information Gain: Selecting samples that are predicted to maximally reduce the overall entropy of the policy distribution after human intervention, thus yielding the greatest learning benefit per sample collected.

3. Expert Divergence: Identifying state-action pairs where multiple expert demonstrations yield significantly different optimal actions, forcing the model to learn a robust consensus policy.

Integrating Human Expertise into Optimization

The Human-in-the-Loop aspect is formalized within E2HiL's objective function, treating human feedback not merely as a reward signal, but as an active component of the optimization landscape. The system maintains an iterative cycle where the current policy pi generates candidate samples, and these candidates are filtered through the entropy guidance mechanism before being presented to the human expert. The paper details how this interaction is modeled:

• The human expert provides corrective actions or preference rankings on a small batch of selected samples.

• These curated samples are then used to refine the policy via an adapted offline RL objective, which minimizes both prediction error and the entropy gap between the current policy and the optimal, human-guided policy.

• This continuous feedback loop allows E2HiL to achieve robust convergence toward expert-level performance with minimal real-world interaction.

Architectural Components and Efficiency Gains

E2HiL is designed as a modular architecture that separates the core RL optimization from the sample selection module. This separation enhances interpretability and allows for targeted improvements in different stages of the learning process. The efficiency gains are quantified by demonstrating that E2HiL significantly outperforms baseline methods (such as standard imitation learning or pure maximum entropy RL) on complex manipulation tasks, achieving state-of-the-art performance with fewer than 30% of the required human interaction time. Furthermore, the framework is shown to be adaptable across various robotic platforms and task domains, validating its utility as a general methodology for safe and efficient real-world deployment.

The primary improvement is the development of a Multi-Modal, Hierarchical, and Calibrated Generalist Agent Architecture (MHCGAA). This architecture moves beyond single-paradigm learning (pure model-free or pure imitation) by integrating the strengths of LLM reasoning, robust world modeling, and highly sample-efficient policy refinement.

This system is designed to operate reliably in unstructured, real-world environments where data collection is expensive and failure carries significant cost.

The MHCGAA comprises four tightly coupled modules:

• Improvement: Integrating a large language model (LLM) as a symbolic reasoning layer above the policy network. The LLM interprets complex, natural language goals (e.g., Clean up the spilled liquid using the vacuum cleaner and place it in the receptacle) and decomposes them into sequential, actionable sub-goals and constraints.

• Mechanism: This module utilizes advanced prompt engineering combined with Retrieval-Augmented Generation (RAG) over procedural manuals or safety protocols to ensure logical consistency before action is taken.

• Citation Basis: Drawing heavily from [31] (LLM enhancement for RL) and [40] (LLM reasoning capabilities).

• Improvement: Implementing a predictive, state-space model that learns the dynamics of the physical environment (t+1 = f(s t, a t)) from heterogeneous data (visual inputs, sensor readings). This model allows for extensive mental rehearsal of complex maneuvers.

• Mechanism: Instead of relying solely on real-time sensory feedback, the agent first simulates a trajectory plan within this learned world model. This significantly reduces the required number of costly physical interactions needed to achieve proficiency (sample efficiency). The system can predict potential failure modes and plan preemptive recovery paths before execution.

• Citation Basis: Building upon [22], [23], and [33] (World Models/Model-Based RL).

• Improvement: Adopting a multi-stage, hybrid learning approach that prioritizes calibration and safety over raw performance maximization.

1. Offline Pre-training: The system first ingests massive amounts of diverse, recorded expert data (human demonstration, simulated failure logs) using calibrated offline RL techniques [44]. This establishes a robust baseline policy.

2. Uncertainty-Aware Fine-Tuning: During limited real-world deployment, the agent continuously estimates its uncertainty regarding its current state and predicted outcomes. It dynamically switches between model-free (exploitation) and model-based (exploration/planning) control based on this quantified uncertainty [23].

3. Human Oversight Loop: Incorporating an interactive, expert-guided loop that allows a human supervisor to correct or guide the agent mid-task, providing immediate, high-value supervisory signals [27].

• Citation Basis: Synthesis of [44] (Calibrated Offline RL), [23] (Uncertainty Awareness), and [27] (Interactive Imitation).

• Improvement: Replacing sparse, binary reward functions with a continuous, self-supervised reward structure that guides exploration intelligently. The system generates intrinsic rewards for achieving information gain or reducing uncertainty about the environment's dynamics, rather than only rewarding goal completion.

• Mechanism: This module actively drives the agent to explore areas of high predictive ambiguity in the world model, ensuring that learning is maximally efficient and robust across novel conditions.

• Citation Basis: Integrating principles from [29] (Exploration-guided shaping) and [30] (Selfsupervised reward shaping).

The MHCGAA can perform complex, multi-step tasks in dynamic, unstructured real-world environments with unprecedented levels of safety, generality, and data efficiency. Specifically:

1. Zero-Shot Task Completion: Given a novel natural language instruction (Please retrieve the red toolbox from the shelf behind the chair and use it to tighten this loose bolt), the system can decompose this goal, plan the entire sequence (locating to traversing to grasping to operating), and execute it without needing prior specific training data for that exact combination of objects or actions.

2. Failure Recovery in Novel States: If a planned action fails due to unforeseen environmental changes (e.g., an object has been moved, or lighting conditions have drastically changed), the system does not halt. It calculates its current state uncertainty, queries its world model for plausible alternative dynamics, and autonomously generates a recovery plan that minimizes deviation from the original objective while maximizing safety.

3. Accelerated Domain Adaptation: The agent can be deployed in an environment with minimal real-world data (e.g., a new factory floor). By leveraging its calibrated offline pre-training and continuous uncertainty monitoring, it achieves performance levels previously requiring hundreds or thousands of hours of expensive physical interaction, dramatically reducing the time-to-deployment.

4. Interactive Collaboration: The system can function as a true collaborator, accepting mid-task human corrections (e.g., No, grab it from the left side instead) and immediately incorporating that corrective data point to update its policy and world model for subsequent actions within the same session.