A Progressive Design Study of Visual Encoders and Value Estimation for Replay-Free Parallelized Q-Learning

arXiv:2608.07335 · cs.LG, cs.AI · Submitted 2026-08-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks".

Jane: The paper was written by Taha Shieenavaz, Shabnam Zareshahraki and Loris Nanni from Department of Information Engineering and University of Padua.

Tom: Stay tuned as we take you through the paper and discuss its implications.

First Look at Aftab: Tom: We're looking at a paper from the University of Padua that's been getting attention in the reinforcement learning community, and I want to make sure we all have the central claim straight before we open the pages. It says the standard three-layer convolutional encoder that model-free RL has been using since 2013 is a bottleneck, and you can get dramatically better results by redesigning it.

Jane: That's exactly the core. The authors, Taha Shieenavaz, Shabnam Zareshahraki, and Loris Nanni, start from the Parallelized Q-Network, PQN, which already made waves by dropping the experience replay buffer and the target network. Then they ask a question nobody had really asked before: what happens if you give that buffer-free learner a proper visual cortex?

Lu: They answer it in three phases. First they benchmark eight different CNN encoders and pick the best balance of performance and efficiency. Then they integrate a representation mechanism called Hadamax, which uses multiplicative feature interactions. Finally they test advanced value heads, dueling, distributional, and ensemble versions, all without a replay buffer.

Meng: And the headline is their final architecture, Aftab. On Atari-57 it reaches an Interquartile Mean human-normalized score of 6 point 479, while the PQN baseline sits at 2 point 692. That's more than doubling the baseline.

Tom: The probability of improvement is 0 point 86, which tells you it's consistent across the fifty-seven games and not a few lucky outliers.

Jane: So what's the bigger read? Lalam, you've been thinking about the architectural side of this field for a while.

Lalam: The bigger read is that vision architectures evolved enormously — residual networks, neural architecture search, transformers — while model-free RL mostly kept a frozen CNN from 2013. This paper treats the encoder as a design variable and gives you a statistically rigorous way to think about it.

Lu: And the parameter discipline makes it convincing. Their chosen backbone, Gamma, has five layers and about 1 point 84 million total parameters. There's a shallow, wide variant called Eta with 23 point 8 million parameters that performs worse. So it's structural depth doing the work, not raw size.

Meng: They're also honest about limits. Aftab's median score of 4 point 42 is well below replay-buffer giants like GDI at 11 point 46 or MuZero at 7 point 31. They don't claim to beat those systems. They claim you can get a lot of performance without the memory overhead, which is a different objective entirely.

Tom: So we've got the thesis, the method, and the numbers. Now we're going to walk through the paper page by page, starting from the title page, and see how they built the case.

The Title Page's Positioning: Tom: Quick check on where we are. We know the paper's promise: a redesign of the encoder in buffer-free RL, with Aftab as the final product. The very first page, the title page, actually does a lot of positioning work for that promise.

Jane: It tells you the date, August 2026, and the target venue, a preprint submitted to Expert Systems with Applications. That venue signals an applied engineering contribution rather than a purely theoretical curiosity.

Lu: The title itself is a commitment. "A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks." Two investigations packed into one title — the encoder and the value head — and the abstract already gives away the headline results, the 6 point 479 IQM and the 0 point 86 probability of improvement.

Meng: The author list is compact, three people in the Department of Information Engineering at the University of Padua. Loris Nanni is a known name in pattern recognition, and the two doctoral students bring the deep reinforcement learning focus.

Tom: The name Aftab, Persian for sunshine, is a deliberate nod to the Rainbow agent, which combined seven algorithmic improvements into one system. They're positioning Aftab as the Rainbow of the buffer-free world.

Jane: And the abstract includes the open-source commitment — the GitHub repository, the model definitions, the raw experimental logs. That kind of transparency sets a tone that carries through the whole paper.

Lalam: The page also tells you the intended audience. This is a recipe paper: here's what works, here's the exact configuration, here's the code. The keywords lay out the ingredient list — distributional RL, deep ensembles, dueling architectures, out-of-distribution generalization.

Lu: And the scope is right there in the abstract too. Fifty-seven Atari games for the main evaluation, sixteen Procgen environments for generalization, four seeds per run, two hundred million frames per configuration.

Meng: So before you even reach the introduction, you know the scale of the evidence. In deep RL, small-scale results are often noise. This paper is built on a volume of computation that makes the claims credible.

Tom: That scale is exactly why the introduction deserves attention, because it frames the gap they're attacking. Let's head to page seven.

The Buffer-Free Machinery: Tom: From the introduction we know the complaint — the field is stuck with a 2013 encoder. Page seven gets into the machinery of the buffer-free approach, and this is where the technical foundations actually get laid.

Jane: The key concept is the Jacobian of the temporal difference update. PQN's stability analysis says the updates stay stable if that Jacobian acts like a contraction, and the threats come from two sources: off-policy instability and nonlinear instability.

Lu: Their fix was LayerNorm to bound the activation norms, plus network width and l2 regularization to control curvature. But here's the twist — this paper deliberately sets l2 to zero, so the normalization and the architecture have to carry the entire stabilizing load on their own.

Meng: The page also credits the parallel infrastructure, EnvPool and JAX, for making this practical. You run 128 environments in parallel, collect synchronous batches, and update directly. No sampling from a giant buffer, no frozen target network, no delayed parameter copies.

Tom: That's a much simpler pipeline, and it leads them to identify the gap this entire paper exploits. Hadamax, the representation mechanism that came after PQN, modified the learning rule, but it inherited the same three-layer CNN body without anyone systematically checking whether that body was the right one.

Jane: Exactly. They say it plainly — the base CNN topological hierarchy was never evaluated. So they're going to evaluate it, controlling parameter counts carefully so they measure structure rather than raw capacity.

Lalam: There's a deeper point embedded here. Computer vision spent years figuring out that depth, skip connections, and receptive fields change what networks can represent. This paper imports that conversation into model-free RL, where the default encoder had gone unquestioned for more than a decade.

Lu: And the receptive field math later in the paper makes the point concrete. The old Nature CNN sees a 36-by-36 patch of the input. Their Gamma-Hadamax encoder sees a 70-by-70 patch, nearly four times the visual context, from a comparable parameter budget.

Meng: So the related work section is a targeted argument in three steps: PQN removed the buffer, EnvPool made it fast, Hadamax updated the learning rule, but the visual feature extractor stayed frozen. That's the opening they drive through.

Tom: And that drives us straight into the methodology, where they define eight encoder variants and a precise way of splitting parameter budgets between convolutional layers and the regression head. Let's keep going.

The Value Toolkit: Tom: We've seen how the encoder question gets set up. Page thirteen switches to the value estimation side and presents a toolbox of three ideas, all of which were originally designed for replay-buffer agents.

Jane: First, dueling networks. Instead of one stream outputting Q-values, you split into a state-value stream and an advantage stream, then subtract the mean of the advantages before combining. That decomposition helps when many actions lead to similar outcomes.

Lu: Second, distributional RL. Rather than predicting the expected return, you predict a whole distribution over returns. The classic C51 formulation uses fifty-one discrete atoms, and learning happens through a KL divergence against a projected target distribution.

Meng: But the paper also brings in a modern twist from the "Stop Regressing" work — reframing value estimation as classification. The scalar Bellman target gets projected onto discrete bins with a two-hot Gaussian scheme, and the network outputs categorical probabilities, so cross-entropy replaces mean squared error.

Tom: That's a meaningful shift. Squared error punishes the size of the mistake, while classification gives smoother gradients and behaves almost like label smoothing. That matters when your targets are noisy temporal difference updates with no replay buffer to average them out.

Jane: And third, bootstrapped exploration. An ensemble of Q-heads sharing one encoder, each originally trained on a different data subset through Bernoulli masking. The idea is to approximate Thompson sampling — commit to one head's policy for a while and explore based on the disagreement between heads.

Lalam: And the detail that matters later is that masking. Bootstrapped DQN used a mask probability like 0 point 5 so each head saw a different random half of the data. This paper keeps the ensemble but sets the mask probability to 1 point 0 — every head sees everything. Diversity comes from random initialization and environment variance alone.

Lu: That's a genuine experiment. Without masking, you'd worry the heads converge to identical functions. Their results show the ensemble still drives deep exploration and improves stability when averaged at evaluation time.

Meng: So page thirteen is the conceptual toolkit: dueling for representation, distributional for target stability, ensembles for exploration. All classic ideas, but all designed on top of replay buffers and target networks.

Tom: And the open question is whether these tools still work when those stabilizing crutches disappear. The next pages show the architectures they built to answer that.

Architectures on the Page: Tom: So we've got the value toolkit from page thirteen. Page nineteen shows the actual architectures, with a figure tracing the evolution from the classic DQN block, through the PQN block with LayerNorm, to the Hadamax block where convolutions stay at stride one and max-pooling handles all the downsampling.

Jane: The Hadamax mechanism itself is an element-wise multiplication of two parallel normalized projections, with GELU activation instead of ReLU. GELU lets small negative values through, which keeps the multiplied gradients from dying during training.

Lu: The table on this page is where the experimental design gets precise. The baseline Hadamax uses big 8-by-8 kernels with aggressive pooling. The Gamma-Hadamax variants keep small 3-by-3 kernels across five blocks, with pooling spaced out more evenly. Same mechanism, completely different hierarchy.

Meng: And the difference between the two Gamma variants is almost absurdly subtle. The Valid version sets pooling padding to zero in layers three and five. The Same version sets it to one. That one padding choice changes the flattened feature map size and doubles the regression head parameters, from 1 point 6 million to 3 point 3 million.

Tom: That's a beautiful controlled experiment. You hold the convolutions constant, change only a padding value, and measure what the extra spatial resolution buys you. The answer, as we'll see later, is nothing — the p-value between them is 0 point 72.

Jane: And the baseline Hadamax control makes the comparison fair. It lands at 4 point 1 million parameters and 163 million FLOPs. Gamma-Hadamax-Valid reaches the same or better performance at 1 point 8 million parameters and 124 million FLOPs. The topology is doing real work.

Lalam: What strikes me is that this is effectively a hand-crafted architecture search guided by hypotheses — depth helps, early downsampling hurts, multiplicative interactions compound with depth. Each variant in the phase tests a specific structural claim.

Lu: And the parameter budgeting from earlier pays off here. Because they separate encoder parameters from head parameters, you can see exactly where bloat lives. For Eta, it's the head. For the Same variant, it's also the head. The encoder stays lean in both.

Meng: So page nineteen leaves us with two structurally similar architectures ready for training. They look nearly identical on paper, and that's precisely the point — tiny structural choices lead to measurable consequences.

Tom: And with the architectures set, the training protocol comes next — including a decision that I suspect raised a few eyebrows. They set weight decay to zero.

Training Without a Crutch: Tom: We're at the training configuration now, and page twenty-five contains the most provocative experimental choice in the entire paper. They set weight decay to exactly zero across every phase and every seed.

Jane: That's a direct break with PQN, which used l2 regularization as part of its mathematical stability argument. The original analysis leans on it to bound the curvature of the nonlinear update and suppress value overestimation.

Lu: And the paper is completely upfront about the break. They devote an entire subsection to it, explaining that removing weight decay lets them isolate the contribution of architecture and normalization from the dampening effect of regularization.

Meng: But they also explicitly decline to claim a theoretical guarantee. No global bound on the TD Jacobian, no formal convergence proof. They say the combination of vectorized sampling, LayerNorm, and their architectural designs maintains operational stability across all tested seeds, and they leave it at that.

Tom: I find that more convincing than a hand-wavy proof would be. They removed the crutch, the system still walks, and here's the evidence across fifty-seven games and four seeds.

Jane: There's a practical angle too. Weight decay is a hyperparameter that somebody has to tune. Removing it removes a tuning knob, which fits this entire philosophy of simplifying the RL pipeline.

Lalam: And it sharpens the causal story. When your models beat the regularized PQN baseline by a large margin, you can't credit clever regularization tuning. The architecture is the explanation.

Lu: It also places the system right in the heart of the deadly triad — function approximation, bootstrapping, and off-policy data — which is famous for causing divergence. The discussion later argues that the gradual channel expansion and the max-pooling downsampling are what keep the system stable.

Meng: The rest of the training recipe is uniform across all experiments: RAdam optimizer, learning rate 2 point 5 times ten to the minus four, two hundred million frames, batch size 4096, 128 parallel environments. Every variant gets the same budget, and that comparability is what makes the architectural comparisons trustworthy.

Jane: One more detail from this section — the loss changes in Phase 3. The first two phases use mean squared error against the λ-return target, while Phase 3 switches to the HL-Gauss two-hot cross-entropy loss for the distributional heads.

Tom: So with training fixed and stability deliberately challenged, we finally hit the results. Phase 1 is all about the eight encoders, and the statistical machinery they bring to the comparison is genuinely impressive.

Phase 1 Results: Tom: Page thirty-one reports Phase 1, the encoder comparison. The headline is that Alpha, a four-layer variant, takes the top IQM at 3 point 536. Gamma, the five-layer variant, is right behind at 3 point 481, while the PQN baseline sits at 2 point 692.

Jane: Both beat the baseline by a large margin. But the paper doesn't crown Alpha, and the justification is statistical — the difference between Alpha and Gamma fails to reach significance under a Wilcoxon signed-rank test with Holm-Bonferroni correction.

Lu: There's also a robustness study. Ten random splits of the fifty-seven games into validation subsets of fifteen, and Gamma wins the efficiency tradeoff every single time. That's a deliberate defense against benchmark overfitting.

Meng: The failures are as informative as the winners. Eta, the shallow wide network with 23 point 8 million parameters, lands at 3 point 114 IQM. Raw capacity in the regression head simply cannot substitute for convolutional depth.

Tom: And Delta, with its giant 9-by-9 first-layer kernel, is significantly worse than the baseline — 2 point 374 IQM and a p-value under 0 point 001. Aggressive early spatial reduction destroys the fine-grained visual information that reactive control needs.

Jane: The probability of improvement matrix is the cleanest summary. Alpha, Beta, and Gamma all sit between 0 point 79 and 0 point 85 against PQN. Delta falls to 0 point 34, and Theta sits at exactly 0 point 50 — no better than a coin flip.

Lalam: That's the right way to read RL results. A single average can hide enormous variance. Pairwise probabilities across the full suite tell you about consistency, and consistency is what you want when choosing an architecture to build on.

Lu: The corrected significance matrices, with 36 pairwise comparisons and the family-wise error rate held at 0 point 05, give you the same message from a different angle. The ordering of the variants isn't noise.

Meng: So Phase 1 establishes the backbone: deeper is better when parameters are controlled, shallow width doesn't help, and giant kernels hurt. Gamma emerges as the balanced foundation, and that sets up Phase 2, where the Hadamax interactions come in.

Tom: And Phase 2 is where the performance really starts to jump. Let's take a look.

Phase 2 Results: Tom: Page thirty-seven opens Phase 2 with a complexity table, and the first thing that jumps out is Gamma-Hadamax-Valid holding the line at 1 point 84 million total parameters — nearly identical to plain Gamma — while the baseline Hadamax control swells to 4 point 1 million.

Jane: The performance gap is decisive. Gamma-Hadamax-Valid hits an IQM of 5 point 325, up from Gamma's 3 point 481, and the Wilcoxon test gives a p-value under 0 point 001. The multiplicative interaction is paying off on top of the deeper topology.

Lu: But the subtle result is the comparison between the two Gamma-Hadamax variants. Same, with the padding that doubles the head parameters to 3 point 5 million, performs no better. The p-value is 0 point 720. Extra capacity, zero measurable benefit.

Meng: The paper explains this through the receptive field analysis. Gamma covers a 39-by-39 patch of the input. The Hadamax variants expand that to 70-by-70

Page 8 of the paper: Tom: So we've just seen the IQM results for the final architecture, and now page 43 puts Aftab on a leaderboard with the big names in Atari.

Jane: That table is a reality check. Aftab's median score of 4 point 42 sits far above the classic DQN at 0 point 79, and it even beats Rainbow's 2 point 31, but it's well short of GDI's 11 point 46 and MuZero's 7 point 31.

Tom: The paper is careful not to overclaim there. They say straight out that those massive scores come from replay-buffer-dependent methods, and competing on raw performance isn't their goal.

Jane: Their goal is to show what a buffer-free agent can do when you actually design the encoder properly. And 4 point 42 median is more than four times human-level performance, which is a strong statement on its own.

Tom: What I like is the framing around memory and throughput. A replay buffer for Atari can eat gigabytes of RAM and create serious I/O bottlenecks during training. Aftab just doesn't have that problem.

Jane: Exactly. They're not saying they're the best agent ever. They're saying they're the best agent you can run without a large memory system, and that's a different trade-off space.

Tom: The caveat section is honest too. They admit the comparison isn't perfectly fair because other methods use different evaluation protocols and different compute budgets.

Jane: And they mention that a full Pareto analysis of wall-clock time, energy, and memory is left for future work. So they know this table is only part of the story.

Tom: But it's the part that makes the paper easy to summarize — a buffer-free agent that beats older replay-based methods and gets respectably close to modern ones.

Jane: So the natural next question is whether those results hold up when the environment itself changes, which is exactly what the Procgen runs test.

Page 9 of the paper: Tom: We've just seen the aggregate IQM scores, and now page 49 lays out the full per-game table that shows exactly where Aftab's gains come from.

Jane: That table is a wild ride. You've got Video Pinball at over a thousand human-normalized, Phoenix at 152, Atlantis at 143, and then you've got games like Demon Attack where Aftab scores just 6 point 9 compared to PQN's 72 point 5.

Tom: So the ensemble and distributional heads aren't universally better — they crush some games and stumble on others.

Jane: Right, and that's precisely why the IQM matters. It chops off the bottom quarter and the top quarter of the results, so those crazy highs and painful lows don't dominate the aggregate.

Tom: Looking at the median tells the same story. Aftab's median is 4 point 42, which is well above human level, but it's much lower than the IQM of 6 point 48 because the middle of the pack is more modest.

Jane: What's interesting is comparing the two finalists here. The Ensemble Dueling head alone gets a median of 3 point 65, and adding the distributional loss in Aftab pushes that to 4 point 42.

Tom: So the distributional piece isn't just a theoretical addition — it genuinely lifts the typical game performance, not just the outliers.

Jane: You also see some games where Aftab actually does worse than plain PQN, like Double Dunk and Skiing going negative. That's a reminder that this isn't a free win everywhere.

Tom: But the fact that the IQM and probability numbers hold up despite those failures is what makes the overall claim credible.

Jane: And that brings us to the next page, where the probability of improvement matrix tells us whether these wins are consistent across the suite or just a few lucky games.

Page 10 of the paper: Tom: We've just seen the full per-game table, and now page 55 shows the learning curves along with the opening of the statistical significance section.

Jane: That figure is actually really telling. It plots IQM across the entire 200 million frames, and then the zoomed panel on the right isolates the final 50 million frames.

Tom: The zoomed panel is where the stability story comes through. Aftab isn't just finishing higher — its line stays flat and high without those late-training dips that plague many RL agents.

Jane: That consistency suggests the ten head ensemble is doing real variance reduction, not just squeezing out a few lucky spikes.

Tom: And right after the figure, the text starts explaining the statistical machinery that backs up those curves.

Jane: They reintroduce the IQM and the Wilcoxon signed-rank test with the Holm-Bonferroni correction, but the new part is how explicitly they argue for filtering the data.

Tom: The IQM chops off the bottom and top quarters of the fifty-seven game scores, so one catastrophic failure or one enormous Video Pinball outlier can't drive the conclusion.

Jane: Exactly. And the pairwise tests across thirty-six comparisons include a strict correction for false positives. The page is saying the curves look good, and here's the formal proof that the differences aren't just noise.

Tom: That's a solid combination — visual evidence coupled with a rigorous statistical framework.

Jane: And with that foundation laid, the paper moves into its limitations, where they'll talk about what this architecture might not handle and where the theory still has gaps.

Conclusion: Tom: So we've walked through the whole Aftab paper, from the encoder benchmark through the Hadamax integration and the final composite architecture, and it's time to wrap up where this leaves the field.

Jane: The big picture is that the paper makes a strong case that the visual encoder has been a silent bottleneck in model-free RL for over a decade, and just replacing that piece can more than double your aggregate score.

Tom: What impressed me most is the discipline. They ran every variant with the same frame budget, the same seeds, the same hyperparameters, and then subjected every comparison to proper multiple-testing corrections.

Jane: And they were honest about the boundaries too. They never claim to beat the replay-buffer giants like GDI or MuZero, and they explicitly say their median score of 4 point 42 is not state-of-the-art.

Tom: Instead, they positioned Aftab as the strongest buffer-free option, which is a different and arguably more practical goal for people working under memory or throughput constraints.

Jane: The removal of weight decay is the boldest empirical claim. The original PQN leaned on that regularization as a mathematical crutch, and they showed the architecture alone can keep training stable across four seeds and fifty-seven games.

Tom: There are real limitations though. Everything is discrete-action, Atari-style, and the Procgen results are much more modest than the Atari gains.

Jane: The Procgen numbers actually make the paper more believable for me. If Aftab had crushed both benchmarks equally, I'd worry about some hidden leakage. Instead, you see a partial transfer, which is typical of RL systems.

Tom: The open-source release with model definitions and raw logs is a big deal for the community, because anyone can build on this without reimplementing from scratch.

Jane: And that's the natural bridge to our next paper, where the authors ask whether these architectural lessons carry over to continuous control, which is exactly the gap they left open.

Department of Information Engineering · University of Padua

cs.LG, cs.AI

Submitted: 2026-08-07

Updated: 2026-09-25

Code: https://github.com/tahashieenavaz/aftab

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 60/100

The gist: This paper, authored by Taha Shieenavaz, Shabnam Zareshahraki, and Loris Nanni from the Department of Information Engineering, University of Padua, Italy, presents a systematic investigation into the

Key concepts

Parallelized Q-Network (PQN)
A reinforcement learning agent that removes the experience replay buffer and target network, using parallel environments and direct updates. Stability is maintained via LayerNorm and network design, rather than traditional stabilization techniques.
Hadamax
A representation mechanism that uses multiplicative feature interactions, combining two parallel normalized projections element-wise with GELU activation. It modifies the learning rule but was originally paired with a standard three-layer CNN, which Aftab replaces.
Interquartile Mean (IQM)
A robust statistical metric for evaluating agent performance across games, averaging the middle 50% of human-normalized scores. It reduces the influence of outliers, providing a reliable comparison between agents.
Dueling and Distributional Value Heads
Value estimation techniques: dueling splits Q-values into state-value and advantage streams, while distributional RL predicts a distribution over returns instead of a single expected value. Both are adapted for buffer-free training in Aftab.

Terminology

Summary

This paper, authored by Taha Shieenavaz, Shabnam Zareshahraki, and Loris Nanni from the Department of Information Engineering, University of Padua, Italy, presents a systematic investigation into the architectural design space of Convolutional Neural Networks (CNNs) for the Parallelized Q-Network (PQN) algorithm. The authors note that "Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm achieves stable off-policy learning without relying on computationally expensive replay buffers or target networks. However, the representational capacity and parameter efficiency of visual encoders operating in these buffer-free settings remain underexplored."

The paper addresses a key gap: while PQN and subsequent representational adaptations like Hadamax have modernized the learning objective; the underlying convolutional backbone remains largely unexamined. The authors argue that by strictly defaulting to legacy feature extractors, the field may be artificially bottlenecking the representational potential of these streamlined algorithms.

The authors enumerate three primary contributions:

  1. Systematic Encoder Evaluation: A benchmarking of eight parameter-constrained CNN topologies within PQN, identifying Gamma as a favorable balance of computational efficiency and performance.

  2. Hadamax Synergies: The architectural integration of the Gamma topology with Hadamard max-pooling mechanisms improves baseline representational capacity.

  3. Advanced Value-Head Ablation: An empirical evaluation of Distributional, Ensemble, and Dueling paradigms applied atop a Hadamax-optimized backbone in a parallelized, replay-free setting.

The final composite model is named Aftab, Persian for sunshine, as a thematic nod to the Rainbow agent, reflecting our strategy of unifying advanced reinforcement learning paradigms into a single entity.

The study was structured into three progressive phases to isolate the impact of topological design from algorithmic capacity.

The authors evaluated eight distinct CNN architectures—denoted Alpha through Theta—within the PQN framework, processing a tensor of four consecutive grayscale environmental frames, stacked to form an input shape of 4 × 84 × 84. Key design considerations included controlling model capacity: we artificially constrain the encoder parameter count in the Beta, Delta, Epsilon, Eta, Zeta, and Theta variants to closely match that of the baseline PQN model (≈ 78k parameters).

Notable variants include:

  • Alpha: closely mirrors the baseline's initial channel counts but adds a fourth layer with 64 channels, nearly doubling encoder parameters to 174,752.

  • Gamma: A five-layer architecture where the channel dimensions are expanded gradually (32→48→64), maintaining a balanced footprint of 1,842,532 total parameters and 24.512 million total FLOPs.

  • Eta: The shallowest model with only two layers but massively expanding the channel width, resulting in a vast feature vector of size 46,208, intentionally driving the total parameter count to ≈ 23.8M.

  • Delta: Utilizes a massive 9 × 9 kernel in the initial layer, which later proved to be a structural pitfall.

  • Theta: The most compact holistic model (≈1.2M total parameters), testing the network's resilience to extreme feature compression.

The Gamma architecture was selected as the optimal backbone despite Alpha achieving a marginally higher IQM HNS (3.536 vs. 3.481). The selection was justified via a robustness validation study with 10 randomized splits of the Atari-57 benchmark, emphasizing a strict performance-complexity tradeoff. A "pairwise Wilcoxon signed-rank comparison between Alpha and Gamma revealed no statistically significant difference in performance following a Holm-Bonferroni correction (p > 0.05). Across all randomized validation splits, this selection rule consistently favored the Gamma architecture."

The authors integrated the Hadamax (Hadamard Max-Pooling) representation mechanism into the Gamma architecture. The Hadamax block replaces spatial strides with max-pooling and introduces multiplicative non-linear interactions via the element-wise Hadamard product (⊙) of two parallel, normalized projections:

zⱼ = MP[f(LN(z j−1W1 j−1)) ⊙ f(LN(z j−1W2 j−1))]

Hadamax also replaces ReLU with GELU, which prevents zero-saturation during the element-wise multiplication, supporting stable gradient flow within the PQN framework.

Three configurations were tested:

  1. Hadamax (Baseline): The control model mimicking the classic three-layer DQN topology but utilizes Hadamax blocks.

  2. Gamma-Hadamax-Valid: Modifies the five-layer Gamma architecture, using pooling with padding of 0 in layers 3 and 5 to act as 'valid' padding.

  3. Gamma-Hadamax-Same: Identical convolutional parameters but introduces a padding of 1 to the pooling operations in layers 3 and 5, preserving spatial resolution.

Complexity profiling showed Gamma-Hadamax-Valid maintains a strictly constrained footprint, requiring only 1,843,556 total parameters and 123.611 million total FLOPs, in stark contrast to the baseline Hadamax which expands the model to 4,125,124 parameters and consuming 162.984 million total FLOPs. Wilcoxon tests revealed Gamma-Hadamax-Valid provides a statistically significant improvement over Gamma (p < 0.001), while the difference between Valid and Same was statistically insignificant (p = 0.720), confirming that preserving spatial resolution through padding in the later layers yields no functional advantage.

The authors provide a mechanistic explanation via receptive field analysis: Gamma achieves a 39×39 effective receptive field, while Gamma-Hadamax expands this to a 70x70 spatial patch. This expansion provides the theoretical mechanism behind the superior Interquartile Mean (IQM) Human-Normalized Scores (HNS) achieved by the Gamma-Hadamax-Valid model.

The final phase evaluated three advanced regression paradigms on the Gamma-Hadamax-Valid backbone, all reconstructed to operate entirely on the synchronous, buffer-free data stream provided by the PQN framework:

  1. Distributional Dueling Head: Combines the dueling architecture with distributional value estimation using N = 51 discrete atoms, spaced uniformly between Vmin = −10.0 and Vmax = 10.0. The approach utilizes Two-Hot Gaussian (HL-Gauss) projection to smooth the target distributions, reframing value estimation entirely as a classification task.

  2. Ensemble Dueling Head: Adapts Bootstrapped DQN into a Deep Ensemble of K=10 independent dueling heads sharing the Gamma-Hadamax-Valid encoder. Crucially, the framework utilizes a mask probability of p = 1.0, meaning every head observes the entirety of the synchronous PQN batch, with diversity maintained strictly through independent random weight initializations and the natural trajectory variance generated across the 128 parallel EnvPool environments. The primary objective is to drive deep exploration by approximating Thompson sampling: each parallel environment uniformly samples its own active head index at the start of an episode to dictate the policy.

  3. Distributional Ensemble Dueling Head: This unified architecture—the final Aftab framework—combines Distributional RL with the Deep Ensemble formulation: The single shared encoder feeds into an ensemble of K = 10 independent heads, where each individual head is a complete Distributional Dueling module outputting A categorical distributions of size N = 51.

A critical deviation from standard PQN is noted: we set the weight decay parameter to exactly 0 across all experimental phases. This design choice is intentional to explicitly isolate the empirical impact of our architectural topologies and multiplicative feature interactions from the dampening effects of parameter regularization. The authors acknowledge: We do not claim a formal convergence guarantee, nor do we assert a global bound on the Temporal Difference (TD) Jacobian for the resulting unregularized, nonlinear updates.

All variants used a unified set of hyperparameters: RAdam optimizer with learning rate 2.5 × 10−4, trained for 200 million environmental frames (50 million steps with frame-skip of 4), discount factor γ = 0.99, GAE λ = 0.65, batch size of 4096, and 128 parallel training environments. Loss functions were MSE for Phases 1 and 2, and HL-Gauss (Two-Hot Gaussian) for Phase 3. All experiments were executed on Nvidia A40 GPUs with four fixed random seeds (1, 2, 3, 4).

The Phase 1 evaluation showed that the Alpha variant achieved an Interquartile Mean (IQM) human-normalized score (HNS) of 3.536, followed by Gamma at 3.481 IQM HNS, both outperforming PQN (Gallici et al., 2024), which is our baseline (2.692 IQM HNS). The deep topological variants demonstrated high probabilities of improvement over the PQN baseline, yielding scores of 0.85, 0.81, and 0.79 for Alpha, Beta, and Gamma respectively. Conversely, the Delta variant, utilizing a large 9 × 9 initial kernel, yielded a severely degraded improvement probability of 0.34 against PQN.

The Eta variant, despite its massive 23.8M parameters, achieved only 3.114 IQM. The pairwise Wilcoxon test between Eta and Zeta yielded a p-value of 0.967, indicating that expanding channel width and linear layer capacity offers negligible functional utility if the underlying spatial feature extractor remains shallow and structurally constrained.

The Hadamax integration substantially improved performance: Gamma-Hadamax-Valid achieved an IQM HNS of 5.343 while restricting its total parameter footprint to 1.84M, outperforming the baseline Hadamax control network (4.12M parameters, 5.149 IQM). The improvement probabilities over the standard Gamma topology were 0.70 and 0.75 for Valid and Same variants respectively.

The final Aftab framework achieved an IQM Human-Normalized Score of 6.479 with a 95% CI of [5.890, 7.210]. The Wilcoxon signed-rank test confirmed a statistically significant performance advantage over the standard Gamma-Hadamax-Valid scalar baseline (5.343 IQM) with p = 0.014. The Probability of Improvement analysis showed the final composite Aftab framework (Distributional Ensemble Dueling) yields a 0.86 probability of improvement over PQN, while the Bootstrapped Dueling configuration achieves a 0.93 probability of improvement over PQN.

The authors report a median HNS of 4.42 for Aftab, noting that competing on unconstrained raw performance is not the objective of this work. We position Aftab as an empirically strong buffer-free alternative for memory-constrained and high-throughput environments.

On the out-of-distribution Procgen Hard benchmark, the final Aftab configuration achieved an aggregate IQM Procgen Normalized Score (PNS) of 0.418 with a 95% CI of [0.132, 0.730], compared to the baseline PQN model, which yielded an IQM PNS of 0.382 (95% CI: [0.189, 0.540]). The authors note that "on environments requiring precise visual tracking amidst high non-stationarity, such as Bigfish (7.233 vs. 31.491) and Dodgeball (0.192 vs. 12.596), the combination of Hadamard representations and ensemble regression heads significantly dampens overfitting to procedurally generated assets."

  1. The Impact of Hierarchical Depth: Both models leverage increased hierarchical depth (four and five layers, respectively) relative to the three-layer baseline architecture, suggesting moderately deeper receptive fields capture spatial dependencies and abstract environmental features more effectively without requiring excessive parameter expansion.

  2. Topology vs. Raw Capacity: The statistical equivalence between Eta and Zeta indicates that expanding channel width and linear layer capacity offers negligible functional utility if the underlying spatial feature extractor remains shallow and structurally constrained.

  3. Penalty of Spatial Bottlenecks: The Delta variant's "significant performance degradation (2.374 IQM) illustrates that aggressive spatial reduction in early layers, specifically via a 9 × 9 initial kernel, correlates with a critical loss of the fine-grained visual granularity required for reactive control tasks. Similarly, Theta (2.649 IQM) demonstrated that halving the output dimensionality in the terminal convolutional layer imposes a representational bottleneck prior to the regression head."

  4. Multiplicative Interactions: Gamma-Hadamax-Valid achieved an IQM HNS of 5.343 while restricting its total parameter footprint to 1.84M. This is less than half the footprint required by the standard three-layer Hadamax control network, establishing that multiplicative interactions compound effectively with optimized convolutional depth, enhancing representation without relying on massive residual stacks.

  5. Buffer-Free Algorithmic Integration: Complex regression heads, which traditionally rely on massive experience replay buffers and target networks to stabilize their loss landscapes, can operate reliably in a strictly online, parallelized regime.

The authors acknowledge several limitations: the observations remain restricted to discrete-action, frame-stacked reinforcement learning paradigms; all evaluations used a single, static set of hyperparameters inherited from the baseline PQN framework; and most importantly, there is the theoretical limitation regarding the absence of formal convergence or stability guarantees for our unregularized network updates. The authors state: we do not provide a closed-form mathematical or Lipschitz bound for this behavior.

The paper concludes that "relying on the historically static 'Nature CNN' backbone restricts algorithmic generalization. By systematically optimizing hierarchical depth, incorporating non-linear multiplicative feature interactions, and deploying variance-reduced, multi-head regression systems, the Aftab framework establishes a stable and computationally efficient architectural reference for parallelized deep reinforcement learning." The complete framework is open-sourced at https://github.com/tahashieenavaz/aftab.

Improvements for AI systems

  • Replace the legacy 3-layer Nature CNN with a systematically optimized 5-layer encoder (Gamma) that expands channels gradually (32 → 48 → 64). This increases hierarchical depth and effective receptive field while keeping total parameters around 1.84M. It raises Atari IQM human-normalized score from PQN's 2.692 to 3.481, without a statistically significant difference from a larger 4-layer encoder.

  • Use Hadamard max-pooling (Hadamax) blocks instead of strided convolutions. Apply two parallel LayerNorm → linear projection → GELU branches, take their Hadamard product, then max-pool with valid padding in layers 3 and 5. This expands the effective receptive field from 39×39 to 70×70, adds multiplicative non-linear feature interactions, and increases IQM HNS from 3.481 to 5.343 while keeping parameters at 1.84M — less than half the baseline Hadamax control.

  • Replace scalar Q-value heads with a Distributional Ensemble Dueling head. Use a shared Gamma-Hadamax encoder followed by K=10 independent heads, each outputting 51 discrete atoms via Two-Hot Gaussian (HL-Gauss) projection over Vmin=-10 to Vmax=10. Keep the ensemble mask probability at 1.0 so diversity comes from independent initialization and natural trajectory variance across 128 parallel environments, not from bootstrapping subsets. This approximates Thompson-sampling-driven exploration without replay buffers or target networks.

  • Avoid known architectural failure modes: do not use a 9×9 initial convolution (Delta: 2.374 IQM, 0.34 probability of improvement), do not use shallow-wide encoders with massive linear layers (Eta ≈ Zeta, p=0.967), and do not halve the terminal convolutional feature map (Theta: 2.649 IQM).

  • Train synchronously and buffer-free with RAdam (lr 2.5e-4), batch size 4096, 128 parallel EnvPool environments, GAE λ=0.65, and 200M environment frames across 4 seeds, evaluating with IQM HNS and Wilcoxon signed-rank tests across randomized Atari splits.

  • Achieve an Atari-57 IQM human-normalized score of 6.479 (95% CI [5.890, 7.210]) — a statistically significant improvement over scalar Gamma-Hadamax (p=0.014) and a 0.86 probability of improvement over the PQN baseline.

  • Operate in memory-constrained, replay-free, target-network-free settings, enabling high-throughput on-policy/off-policy learning with only synchronous parallel trajectory streams.

  • Better generalize to out-of-distribution procedural environments — aggregate Procgen Hard IQM of 0.418 vs. 0.382 for PQN, with significantly reduced overfitting on fast-moving procedural assets like Bigfish and Dodgeball.

  • Run with a compact encoder footprint (1.84M encoder parameters, 123.6M FLOPs), making it suitable for deployment where storage/compute per update is limited.

  • Provide a reusable, open-sourced architecture (Aftab) for building stable, parallelized deep RL agents that integrate hierarchical convolutional encoders, multiplicative representation mechanisms, and distributional ensemble value heads.

Abstract

Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm enables off-policy value learning without relying on experience replay buffers or target networks. However, the representational capacity and computational efficiency of visual encoders operating in these buffer-free settings remain comparatively underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks within PQN. We evaluate eight distinct CNN topologies while explicitly characterizing their parameter and computational requirements. We further study the effect of multiplicative representation learning and advanced value estimation by integrating the Hadamax encoding paradigm with categorical, ensemble, and dueling value heads. Extensive experiments on Atari-57 show that our final composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.592, compared with 2.715 for the standard PQN baseline, together with a 0.86 Probability of Improvement over PQN. We additionally evaluate Aftab on Procgen-Hard to assess performance under procedurally varying visual environments. Aftab achieves a normalized learning-curve Area Under the Curve (nAUC) of 0.541 compared with 0.216 for PQN. Overall, the results demonstrate that carefully designed encoder topology, multiplicative feature interactions, and advanced value-estimation heads can substantially improve performance within a parallelized, replay-free Q-learning framework while preserving its memory-efficient training paradigm. The complete Aftab framework, including model definitions, training configurations, reproducibility settings, and raw experimental logs, is open-sourced at https://github.com/tahashieenavaz/aftab

Sources

Related papers