A nonlinear hidden layer enables actor-critic agents to learn multiple paired association navigation

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "A nonlinear hidden layer enables actor-critic agents to learn multiple paired association navigation".

Marcus: Here's the gist of the paper. The gist Navigation to multiple cued reward locations has been increasingly used to study rodent learning, and this computational study shows that adding a nonlinear hidden layer enables actor-critic agents to learn multiple paired association navigation <ref:2106.13541#pg3> How it works The core limitation of classic actor-critic agents is their inability to learn multiple paired association navigation, which is overcome by an agent in which place cell and cue information are first processed by a feedforward nonlinear hidden layer with synapses to the actor and critic subject to temporal difference error-modulated plasticity <ref:2106.13541#pg4> Faster learning is obtained when the feedforward layer is replaced by a recurrent reservoir network <ref:2106.13541#pg3> Learning single reward locations The agents were verified to learn the single reward location task, and this was demonstrated by the decrease in latency in reaching the single reward location over 42 trials <ref:2106.13541#pg8> The Classic agent, Expanded Classic agent, Linear Hidden Layer agent, and Nonlinear Hidden Layer agent all learned to navigate to a single reward location comparably well <ref:2106.13541#pg9> In a probe trial, no reward was given even if the agent reached the correct reward location <ref:2106.13541#pg10> Learning to navigate to a displaced reward location The agents were investigated for their ability to adapt to displacement of the reward location, and this variant of the single reward location task can be learned by biologically plausible reinforcement learning agents For the case in which the reward location was not displaced, all agents with plasticity continued to increase the amount of time spent near the reward location from PT3 (Fig.

Ines: 1D) to PT6 For the displaced reward locations, all agents spent increasing amounts of time near the reward location from PT4 to PT6 The higher performance with more trials for Reward Locations 1–4 is reflected in example Classic and Nonlinear Hidden Layer agents having value maps with higher values more concentrated near the displaced reward location, and policy maps that were more directed toward the displaced reward location in later probe trials Learning multiple paired association navigation Only the Nonlinear Hidden Layer agent was able to learn a multiple paired association navigation task when the sensory cue indicating the reward location is present throughout each trial The Nonlinear Hidden Layer agent's latency decreased to 13 seconds, while the Classic, Expanded Classic and Linear Hidden Layer agents' latencies plateaued at 110 seconds The Nonlinear Hidden Layer agent showed above chance (p < 0.0001) visit ratios in all probe sessions, and improved from PS1 to PS2 to PS3 In contrast, an example Nonlinear Hidden Layer agent moved more directly to the correct reward location for each cue The Classic, Expanded Classic, and Linear Hidden Layer agents did not exhibit above chance performance on the multiple paired association task Critical hyperparameters for learning multiple paired associates The study examined the effect of hyperparameters that affects hidden layer output dimensionality, including the number of hidden units, hidden unit activation function, and distribution of excitatory and inhibitory synaptic weights onto hidden units Increasing the expansion ratio from 1 to 175 increased the average number of cue-reward associations learned from 0.02 ± 0.14 (SD) to 14.9 ± 0.9 (SD) The ReLU nonlinearity allowed 14.9 ± 0.9 (SD) associations to be learned on average Integrating working memory for learning multiple PAs Adding a bump attractor to the agent provided working memory that enabled the task to be learned when the cue was present only at the start of each trial We also showed that using a reservoir of recurrently connected neurons for the hidden layer improved learning, and allowed agents to learn as quickly as animals The Reservoir agents learned faster than the Nonlinear Hidden Layer agents, showing an advantage of a recurrent reservoir over a feedforward layer Continuous temporal difference error and synaptic plasticity The continuous TD error was defined as (t) = r(t) + v̇(t) − 1/τg v(t) Synaptic plasticity of the weights onto the actor are governed by a 3-factor rule, being modulated by the continuous TD error, the presynaptic firing rate, and the postsynaptic firing rate Synaptic plasticity of the weights onto the critic are governed by a 2-factor rule, being modulated by the continuous TD error and the presynaptic firing rate Generation of value and policy maps Spatial value maps were generated from the critic's firing rate, and spatial policy maps were generated from the actor's action The mean critic firing rate at each bin was computed for the duration of the cue-probe trial over all iterations and visualized as a heatmap The vector sum of the action at each bin was computed for the duration of the cue-probe trial over all iterations and visualized as a quiver plot Data availability Code for our results is available at https://github.

Title and authors: Marcus: Let's talk about the title again, "A nonlinear hidden layer enables actor-critic agents to learn multiple paired association navigation". It sounds very technical, but it’s pointing directly at the architectural fix they found for this specific learning problem.

Ines: Yeah, the authors are Ganesh Kumar and his collaborators. They are building on older work about how these agents navigate to a single reward location, and they’re showing that by adding a nonlinear hidden layer, you unlock the ability to learn multiple paired associations.

Yuki: I wonder if this means that for complex tasks in nature, we might need this kind of layered processing instead of just simple feedforward connections between place cells and the motor output.

Marcus: That’s a good thought. The paper suggests that by having this hidden layer, you can handle more associations than before. It moves beyond the limitations of the classic agent structure we talked about earlier.

Ines: Precisely. The authors are showing that they can learn multiple paired association navigation when the sensory cue indicating the reward location is present throughout each trial, which is a key condition for testing their ability to handle these multiple targets correctly.

The paper's summary: Ines: So, what does the summary actually tell us about how this works? It seems they’re describing a system where the place cell and cue information go through this nonlinear hidden layer before hitting the actor and critic networks.

Marcus: Right. And they mention that there's temporal difference error-modulated plasticity involved, which is how the connections actually change based on what happened during the trial—the reward or not.

Yuki: That sounds like a biologically plausible mechanism for updating those associations over time, which is crucial if we want this computational model to reflect real biological learning processes.

Ines: Exactly. They describe how the value function and reward information are used to calculate a temporal difference error, denoted as delta t, and that this error then modulates the synaptic plasticity of the actor's weights in a specific way.

Marcus: And they also mention that they found faster learning when you replaced that feedforward layer with something called a recurrent reservoir network instead. That’s an interesting alternative mechanism they explored.

Yuki: So, it’s not just one solution; they are comparing the impact of different internal structures on how effectively the agent can learn these multiple tasks. It gives us a better picture of what might be necessary for robust learning.

The paper's improvements: Marcus: Now, let's talk about what they suggest as improvements over the existing work in this paper, because this isn't just a baseline; it’s an improvement on previous methods. They show how this nonlinear structure helps them learn multiple pairs robustly.

Ines: The main improvement they highlight is that only the Nonlinear Hidden Layer agent was able to learn a multiple paired association navigation task when the sensory cue indicating the reward location is present throughout each trial, and it showed above chance performance in visit ratios.

Yuki: That’s a big jump from where other agents like the Classic and Linear Hidden Layer agents were at; they just couldn't get that high enough visit ratio. It suggests that this nonlinear layer is what provides the necessary representational power for this kind of task.

Ines: Right. And they also looked into hyperparameter tuning, showing that increasing the expansion ratio from one to one hundred seventy-five really increased how many cue-reward associations the system learned, hitting an average of fourteen point nine associations when using a ReLU activation function.

Marcus: So, it’s not just about one structural change; it’s about tuning the specific settings of that layer too. The paper points to the distribution of excitatory and inhibitory synaptic weights as critical for achieving this high number of learned associations.

Conclusion: Ines: So, to wrap things up, the main point is that adding this nonlinear hidden layer architecture allows actor-critic agents to learn multiple paired association navigation when the cue is consistently present. It also shows that tuning parameters like the expansion ratio and using a ReLU activation function lets you learn quite a few associations.

Marcus: And they also explored ways to improve learning speed by using a recurrent reservoir network instead of just a simple feedforward layer, finding that the reservoir agents learned faster in some scenarios.

Yuki: From my viewpoint, what this means for the wider field is that it gives us a more concrete blueprint for how complex sensory inputs are integrated into decision-making systems when those inputs have multiple relevant targets.

Ines: So, this paper, "A nonlinear hidden layer enables actor-critic agents to learn multiple paired association navigation", demonstrates that a specific type of network structure can unlock new capabilities for reinforcement learning agents beyond simple single-task navigation.