Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation

The paper introduces EDPFRL-IM (Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation), a novel framework that addresses a critical gap in Personalized Federated Reinforcement Learning (PFRL). The authors note that "many current methods for PFRL rely heavily on exploiting existing reinforcement learning reward signals to derive an optimal policy for each client, thereby neglecting exploration in non-stationary or sparse-reward environments. The proposed framework leverages an inherent curiosity-driven exploration at each client to promote local exploration and protect client privacy." The framework uses Random Network Distillation (RND) signals added to extrinsic rewards to facilitate policy discovery in unexplored state spaces. The server maintains privacy by not accessing clients' raw experiences or local gradient estimates; instead, it sends global exploration priors and collects minimal novelty summaries from each client to enable both diverse and coordinated exploration among clients. Experiments demonstrate that EDPFRL-IM outperforms average PFRL benchmarks in policy personalization and sample efficiency, primarily in delayed and sparse reward systems.

The paper establishes that Reinforcement learning (RL) is a powerful framework for sequential decision-making that enables agents to learn optimal behaviors through interaction with their environment. Federated Reinforcement Learning (FRL) enables multiple clients to cooperatively train reinforcement learning (RL) agents in decentralized settings without exchanging raw experience data. Personalized FRL (PFRL) expands on this by customizing policies to each client's particular environment and objectives, making it suitable for applications such as health monitoring and robotics.

The authors identify a key limitation: Although existing PFRL frameworks focus on policy optimization and aggregation, they often overlook the role of exploration under limited or delayed rewards. Standard RL strategies like ϵ-greedy or entropy regularization become less effective in federated settings, where clients operate independently and face privacy constraints that limit coordination. While approaches like FedRL, FedAvg-RL, and pFedMe address personalization or aggregation, they assume homogeneous or supervised settings. Intrinsic motivation methods such as RND, ICM, and count-based exploration show promise for centralized RL but remain underexplored in federated scenarios.

The paper summarizes three main contributions:

1. Introduction of EDPFRL-IM: a novel FRL framework that incorporates intrinsic motivation for personalized exploration in a decentralized setting.

2. Communication-efficient protocol: for compressing exploration statistics, enabling clients to align their curiosity-driven behavior while avoiding disclosure of raw data or policies.

3. Empirical evaluation: in sparse-reward benchmark environments and demonstrate significant improvements in personalization, exploration efficiency, and policy performance over existing federated RL baselines.

The framework considers a PFRL setting with N clients, each interacting with a local Markov decision process (MDP) Mi = (Si, A, Pi, Ri, γ), where Si and Ri are client-specific, while A and γ are shared. Due to heterogeneity, each client learns a personalized policy πi(as) to maximize its expected return as defined by the objective function Ji(πi) = Eπi[Σ γ t ri,t]. Clients update policies via gradient ascent: θi ← θi + η∇θi Ji(πθi), with learning rate η.

The framework adopts Random Network Distillation (RND) as the intrinsic motivation mechanism. Each client uses a fixed random target network ftgt and a trainable predictor fpred. The intrinsic reward is calculated as the prediction error on the next state st+1: rtint = ∥ftgt(st+1) − fpred(st+1)∥2. This promotes the exploration of novel states. The total reward combines extrinsic and intrinsic terms as rttotal = rtext + αi · rtint, where αi controls the exploration-exploitation trade-off per client.

To address the problem that uncoordinated exploration can lead to redundant efforts across clients, the paper proposes a federated privacy-preserving coordination mechanism. Each client generates compact exploration statistics Ei, such as visitation counts, frequency histograms over state embeddings, or top-k novel state hashes based on intrinsic rewards. The server aggregates these into a global exploration profile: G = Aggregate(Ei Ni=1). The global novelty prior is computed as Pnovel(s) = 1/(1 + vglobal[cluster(s)]), where cluster(s) maps states to cluster indices.

Clients incorporate the global novelty prior into policy updates with the sampling probability: pi(s) ∝ (1 + β · Pnovel(s)) · puniform i(s), with puniform i(s) = 1/Di as the uniform baseline probability and β ≥ 0 controlling the influence of the global prior.

The framework was tested in standard reinforcement learning environments with sparse rewards and heterogeneous dynamics, specifically modified versions of MountainCar-v0 and CartPole-sparse, which are chosen for their sensitivity to exploration quality and their significance in comparing sample efficiency.

The simulation considers N = 10 clients, each running in a different local environment built from MountainCar-v0 and a modified CartPole environment with sparse rewards. Heterogeneity is created by adjusting the elements of the environment, including gravity, friction, and reward shaping. For example, in MountainCar-v0, clients earn additional shaping incentives based on distance or velocity, and gravity varies from 0.0025 to 0.006. These differences lead to different MDPs, indicating extremely non-IID conditions where effective policy learning requires personalization.

The approach is compared against: "(i) Local RL, in which every client trains on its own without any federated coordination; (ii) FedRL, a common FRL method with global policy aggregation; and (iii) FedRL+RND, which applies intrinsic rewards locally without exploration coordination. Additional baselines include single-agent intrinsic-motivation methods run in the non-federated setting (RND, ICM, and count-based exploration (CBE)), as well as federated and personalized FL baselines without coordinated exploration (FedAvg-RL, pFedMe, and FedPer++)."

Training proceeds for T = 100 global communication rounds, with each client performing E = 10 local PPO updates per round. Policies use fully connected two-layer neural networks with 64 hidden units and ReLU activations. Each client has its own Random Network Distillation (RND) module, with both the target and predictor networks consisting of a single hidden layer of size 128. The intrinsic reward coefficient αi is set to 0.1, while the novelty sampling bias β is set to 0.5.

The results show that EDPFRL-IM outperformed all baselines (Local RL, FedRL, and FedRL+RND) consistently throughout the initial rounds, with significantly higher returns due to the efficient use of available samples. In MountainCar-v0, EDPFRL-IM achieved quick convergence with a smooth learning curve despite the presence of sparse rewards and difficult state transition dynamics. In CartPole-sparse, EDPFRL-IM had a similar advantage over the other methods. When comparing to FedRL+RND, EDPFRL-IM produced greater average returns and exhibited a more consistent rate of convergence, supporting the claim that coordinated exploration through aggregated novelty priors contributed to superior performance.

Table I shows the mean returns for each client from both MountainCar-v0 and CartPole-sparse, with EDPFRL-IM producing greater performance than the FedRL baseline across all clients. For MountainCar-v0, EDPFRL-IM returns range from 0.66 to 0.75 across clients, while FedRL ranges from 0.38 to 0.44. For CartPole-sparse, EDPFRL-IM ranges from 0.65 to 0.72, while FedRL ranges from 0.33 to 0.38. The authors note that the lower performance variance resulting from EDPFRL-IM allows for greater consistency and personalization of policy learning.

Table II presents an ablation study examining the impact of exploration coordination. Key findings include:

• When intrinsic motivation is completely removed, as with the FedRL implementation, agent performance is low due to the lack of exploratory behavior (0.40 in MountainCar, 0.35 in CartPole).

• Adding only the intrinsic reward component (FedRL+RND) provides some improvement but lacks clear directionality (0.60 in MountainCar, 0.54 in CartPole).

• Removing the global novelty prior from EDPFRL-IM (i.e., β = 0) improves performance slightly through individualized customization (0.63 in MountainCar, 0.58 in CartPole).

• EDPFRL-IM achieves the best returns in both environments when coordinated exploratory behavior is used (i.e., β = 0.5) (0.80 in MountainCar, 0.76 in CartPole).

Figure 3 shows that clients' exploration coverage (percentage of the state space that clients visit) in both environments is much greater when using EDPFRL-IM compared to using FedRL and FedRL+RND. This indicates that the coordinated use of intrinsic motivation between clients is crucial for increasing the diversity of exploration.

The paper assessed the adaptability of the three methods to the introduction of a cold-start client during training (at round 5). Results show that "EDPFRL-IM uses globally aggregated exploration knowledge to facilitate rapid and smooth adaptation for cold-start clients in both environments, whereas FedRL and FedRL+RND do not use this form of globally aggregated knowledge, which negatively affects their ability to support early-stage learners, resulting in slower progress."

Table III presents the final average return achieved by several federated and personalized reinforcement learning algorithms. EDPFRL-IM achieved 0.76 in MountainCar-v0 and 0.74 in CartPole-sparse, outperforming:

• RND: 0.31 / 0.32

• ICM: 0.34 / 0.36

• CBE: 0.39 / 0.49

• FedAvg-RL: 0.38 / 0.42

• pFedMe: 0.40 / 0.45

• FedPer++: 0.41 / 0.44

• FedRL+RND: 0.48 / 0.52

The authors note that "although approaches such as pFedMe and FedPer++ support personalization of RL models, they provide limited exploration capabilities for discovering new behaviors. FedRL+RND incorporates intrinsic motivation through curiosity-driven exploration but lacks an efficient mechanism for personalized federated adaptation across clients."

The paper concludes that "EDPFRL-IM addresses an important shortcoming of prior, less effective PFRL systems by incorporating a global novelty prior computed from a compressed set of client exploration statistics, enabling each client to learn while adapting to its local context and benefiting from the collective experience of the remaining clients. The comprehensive evaluation demonstrates that it achieves significant improvements in sample efficiency and personalization, and adapts faster for cold-start clients than its peers. The framework consistently outperforms baseline systems using both standard and personalized FL algorithms, including various intrinsic-incentive-based FL algorithms, and does so with no compromise in terms of privacy."

Improvements to AI Systems:

1. Privacy-Preserving Coordinated Exploration via Compressed Novelty Summaries

• AI systems can now share exploration insights across distributed agents without exposing raw experiences or gradients. Each agent sends compact statistics (e.g., state visitation counts, top-k novel state hashes) to a central server, which aggregates them into a global novelty prior. This enables agents to avoid redundant exploration and focus on globally under-explored regions, while maintaining strict privacy guarantees.

2. Curiosity-Driven Personalization with Adaptive Exploration-Exploitation Trade-offs

• Each agent uses Random Network Distillation (RND) to generate intrinsic rewards based on prediction error, but the weight of these rewards (αi) is personalized per client. This allows the system to automatically adjust how much each agent explores vs. exploits based on its local environment’s reward sparsity or non-stationarity, improving sample efficiency in heterogeneous settings.

3. Global Novelty Prior for Rapid Cold-Start Adaptation

• New agents joining the system can immediately leverage the aggregated global exploration knowledge (e.g., which state clusters are already well-known) to bootstrap their own exploration. This reduces the time and samples needed for cold-start clients to reach competent performance, as demonstrated by the paper’s robustness tests.

4. Communication-Efficient Exploration Coordination

• The framework compresses exploration statistics into minimal summaries (e.g., cluster-level histograms), reducing communication overhead compared to sharing full policies or raw trajectories. This makes coordinated exploration feasible in bandwidth-constrained federated settings.

5. Improved Sample Efficiency in Sparse-Reward Environments

• By combining intrinsic motivation with federated coordination, the system achieves higher returns with fewer environment interactions. For example, in MountainCar-v0 and CartPole-sparse, the method reaches 0.76–0.80 average returns versus 0.38–0.52 for baselines, demonstrating faster convergence and better final performance.

6. Robust Personalization Under Non-IID Client Dynamics

• The system learns a personalized policy per client while still benefiting from shared exploration priors. This is critical for applications like health monitoring or robotics where each client has different dynamics (e.g., varying gravity, friction, or reward shaping). The method reduces performance variance across clients, ensuring consistent quality.

7. Seamless Integration with Existing RL Algorithms

• The framework is agnostic to the underlying RL optimizer (e.g., PPO used in experiments). It can be layered onto any policy-gradient method, making it easy to deploy in existing federated RL pipelines without major architectural changes.

What the Improved AI System Can Do:

• Deploy in heterogeneous, privacy-sensitive environments (e.g., hospital networks, smart home systems, or robot fleets) where agents must learn personalized behaviors without sharing sensitive data.

• Achieve high performance in sparse-reward tasks (e.g., navigation, manipulation, or recommendation systems with delayed feedback) by efficiently exploring state spaces that would otherwise be ignored.

• Adapt quickly to new users or devices by using global exploration knowledge, reducing the “cold-start” problem in personalized services.

• Scale to large federated networks with limited communication bandwidth, as only compact summaries are exchanged.

• Maintain consistent performance across diverse clients even when their environments differ significantly, avoiding the pitfalls of one-size-fits-all policies.

Personalized Federated Reinforcement Learning (PFRL) takes a decentralized approach to storing and accessing information based on past experiences while keeping each client's data private during the learning of each client's policy. Many current methods for PFRL rely heavily on exploiting existing reinforcement learning reward signals to derive an optimal policy for each client, thereby neglecting exploration in non-stationary or sparse-reward environments. In this work, we introduce a new exploration-driven framework, Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation (EDPFRL-IM), that leverages an inherent curiosity-driven exploration at each client to promote local exploration and protect client privacy. Furthermore, to facilitate policy discovery via exploration in previously unexplored state spaces, clients add an intrinsic random network distillation (RND) signal to their extrinsic reward. Additionally, the server does not have access to clients' raw experiences or local gradient estimates; instead, the server sends global exploration priors and collects minimal novelty summaries from each client to enable both diverse and coordinated exploration among clients. Experiments in benchmark environments show that our framework outperforms average PFRL benchmarks in policy personalization and sample efficiency, primarily in delayed and sparse reward systems. Overall, EDPFRL-IM enables the integration of a flexible exploratory learning structure into federated reinforcement learning systems while preserving client privacy.