2608.07335-Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks".

Jane: The paper was written by Taha Shieenavaz, Shabnam Zareshahraki and Loris Nanni from Department of Information Engineering and University of Padua.

Tom: Stay tuned as we take you through the paper and discuss its implications.

First Look at Aftab: Tom: We're looking at a paper from the University of Padua that's been getting attention in the reinforcement learning community, and I want to make sure we all have the central claim straight before we open the pages. It says the standard three-layer convolutional encoder that model-free RL has been using since 2013 is a bottleneck, and you can get dramatically better results by redesigning it.

Jane: That's exactly the core. The authors, Taha Shieenavaz, Shabnam Zareshahraki, and Loris Nanni, start from the Parallelized Q-Network, PQN, which already made waves by dropping the experience replay buffer and the target network. Then they ask a question nobody had really asked before: what happens if you give that buffer-free learner a proper visual cortex?

Lu: They answer it in three phases. First they benchmark eight different CNN encoders and pick the best balance of performance and efficiency. Then they integrate a representation mechanism called Hadamax, which uses multiplicative feature interactions. Finally they test advanced value heads, dueling, distributional, and ensemble versions, all without a replay buffer.

Meng: And the headline is their final architecture, Aftab. On Atari-57 it reaches an Interquartile Mean human-normalized score of 6 point 479, while the PQN baseline sits at 2 point 692. That's more than doubling the baseline.

Tom: The probability of improvement is 0 point 86, which tells you it's consistent across the fifty-seven games and not a few lucky outliers.

Jane: So what's the bigger read? Lalam, you've been thinking about the architectural side of this field for a while.

Lalam: The bigger read is that vision architectures evolved enormously — residual networks, neural architecture search, transformers — while model-free RL mostly kept a frozen CNN from 2013. This paper treats the encoder as a design variable and gives you a statistically rigorous way to think about it.

Lu: And the parameter discipline makes it convincing. Their chosen backbone, Gamma, has five layers and about 1 point 84 million total parameters. There's a shallow, wide variant called Eta with 23 point 8 million parameters that performs worse. So it's structural depth doing the work, not raw size.

Meng: They're also honest about limits. Aftab's median score of 4 point 42 is well below replay-buffer giants like GDI at 11 point 46 or MuZero at 7 point 31. They don't claim to beat those systems. They claim you can get a lot of performance without the memory overhead, which is a different objective entirely.

Tom: So we've got the thesis, the method, and the numbers. Now we're going to walk through the paper page by page, starting from the title page, and see how they built the case.

The Title Page's Positioning: Tom: Quick check on where we are. We know the paper's promise: a redesign of the encoder in buffer-free RL, with Aftab as the final product. The very first page, the title page, actually does a lot of positioning work for that promise.

Jane: It tells you the date, August 2026, and the target venue, a preprint submitted to Expert Systems with Applications. That venue signals an applied engineering contribution rather than a purely theoretical curiosity.

Lu: The title itself is a commitment. "A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks." Two investigations packed into one title — the encoder and the value head — and the abstract already gives away the headline results, the 6 point 479 IQM and the 0 point 86 probability of improvement.

Meng: The author list is compact, three people in the Department of Information Engineering at the University of Padua. Loris Nanni is a known name in pattern recognition, and the two doctoral students bring the deep reinforcement learning focus.

Tom: The name Aftab, Persian for sunshine, is a deliberate nod to the Rainbow agent, which combined seven algorithmic improvements into one system. They're positioning Aftab as the Rainbow of the buffer-free world.

Jane: And the abstract includes the open-source commitment — the GitHub repository, the model definitions, the raw experimental logs. That kind of transparency sets a tone that carries through the whole paper.

Lalam: The page also tells you the intended audience. This is a recipe paper: here's what works, here's the exact configuration, here's the code. The keywords lay out the ingredient list — distributional RL, deep ensembles, dueling architectures, out-of-distribution generalization.

Lu: And the scope is right there in the abstract too. Fifty-seven Atari games for the main evaluation, sixteen Procgen environments for generalization, four seeds per run, two hundred million frames per configuration.

Meng: So before you even reach the introduction, you know the scale of the evidence. In deep RL, small-scale results are often noise. This paper is built on a volume of computation that makes the claims credible.

Tom: That scale is exactly why the introduction deserves attention, because it frames the gap they're attacking. Let's head to page seven.

The Buffer-Free Machinery: Tom: From the introduction we know the complaint — the field is stuck with a 2013 encoder. Page seven gets into the machinery of the buffer-free approach, and this is where the technical foundations actually get laid.

Jane: The key concept is the Jacobian of the temporal difference update. PQN's stability analysis says the updates stay stable if that Jacobian acts like a contraction, and the threats come from two sources: off-policy instability and nonlinear instability.

Lu: Their fix was LayerNorm to bound the activation norms, plus network width and ℓ2 regularization to control curvature. But here's the twist — this paper deliberately sets ℓ2 to zero, so the normalization and the architecture have to carry the entire stabilizing load on their own.

Meng: The page also credits the parallel infrastructure, EnvPool and JAX, for making this practical. You run 128 environments in parallel, collect synchronous batches, and update directly. No sampling from a giant buffer, no frozen target network, no delayed parameter copies.

Tom: That's a much simpler pipeline, and it leads them to identify the gap this entire paper exploits. Hadamax, the representation mechanism that came after PQN, modified the learning rule, but it inherited the same three-layer CNN body without anyone systematically checking whether that body was the right one.

Jane: Exactly. They say it plainly — the base CNN topological hierarchy was never evaluated. So they're going to evaluate it, controlling parameter counts carefully so they measure structure rather than raw capacity.

Lalam: There's a deeper point embedded here. Computer vision spent years figuring out that depth, skip connections, and receptive fields change what networks can represent. This paper imports that conversation into model-free RL, where the default encoder had gone unquestioned for more than a decade.

Lu: And the receptive field math later in the paper makes the point concrete. The old Nature CNN sees a 36-by-36 patch of the input. Their Gamma-Hadamax encoder sees a 70-by-70 patch, nearly four times the visual context, from a comparable parameter budget.

Meng: So the related work section is a targeted argument in three steps: PQN removed the buffer, EnvPool made it fast, Hadamax updated the learning rule, but the visual feature extractor stayed frozen. That's the opening they drive through.

Tom: And that drives us straight into the methodology, where they define eight encoder variants and a precise way of splitting parameter budgets between convolutional layers and the regression head. Let's keep going.

The Value Toolkit: Tom: We've seen how the encoder question gets set up. Page thirteen switches to the value estimation side and presents a toolbox of three ideas, all of which were originally designed for replay-buffer agents.

Jane: First, dueling networks. Instead of one stream outputting Q-values, you split into a state-value stream and an advantage stream, then subtract the mean of the advantages before combining. That decomposition helps when many actions lead to similar outcomes.

Lu: Second, distributional RL. Rather than predicting the expected return, you predict a whole distribution over returns. The classic C51 formulation uses fifty-one discrete atoms, and learning happens through a KL divergence against a projected target distribution.

Meng: But the paper also brings in a modern twist from the "Stop Regressing" work — reframing value estimation as classification. The scalar Bellman target gets projected onto discrete bins with a two-hot Gaussian scheme, and the network outputs categorical probabilities, so cross-entropy replaces mean squared error.

Tom: That's a meaningful shift. Squared error punishes the size of the mistake, while classification gives smoother gradients and behaves almost like label smoothing. That matters when your targets are noisy temporal difference updates with no replay buffer to average them out.

Jane: And third, bootstrapped exploration. An ensemble of Q-heads sharing one encoder, each originally trained on a different data subset through Bernoulli masking. The idea is to approximate Thompson sampling — commit to one head's policy for a while and explore based on the disagreement between heads.

Lalam: And the detail that matters later is that masking. Bootstrapped DQN used a mask probability like 0 point 5 so each head saw a different random half of the data. This paper keeps the ensemble but sets the mask probability to 1 point 0 — every head sees everything. Diversity comes from random initialization and environment variance alone.

Lu: That's a genuine experiment. Without masking, you'd worry the heads converge to identical functions. Their results show the ensemble still drives deep exploration and improves stability when averaged at evaluation time.

Meng: So page thirteen is the conceptual toolkit: dueling for representation, distributional for target stability, ensembles for exploration. All classic ideas, but all designed on top of replay buffers and target networks.

Tom: And the open question is whether these tools still work when those stabilizing crutches disappear. The next pages show the architectures they built to answer that.

Architectures on the Page: Tom: So we've got the value toolkit from page thirteen. Page nineteen shows the actual architectures, with a figure tracing the evolution from the classic DQN block, through the PQN block with LayerNorm, to the Hadamax block where convolutions stay at stride one and max-pooling handles all the downsampling.

Jane: The Hadamax mechanism itself is an element-wise multiplication of two parallel normalized projections, with GELU activation instead of ReLU. GELU lets small negative values through, which keeps the multiplied gradients from dying during training.

Lu: The table on this page is where the experimental design gets precise. The baseline Hadamax uses big 8-by-8 kernels with aggressive pooling. The Gamma-Hadamax variants keep small 3-by-3 kernels across five blocks, with pooling spaced out more evenly. Same mechanism, completely different hierarchy.

Meng: And the difference between the two Gamma variants is almost absurdly subtle. The Valid version sets pooling padding to zero in layers three and five. The Same version sets it to one. That one padding choice changes the flattened feature map size and doubles the regression head parameters, from 1 point 6 million to 3 point 3 million.

Tom: That's a beautiful controlled experiment. You hold the convolutions constant, change only a padding value, and measure what the extra spatial resolution buys you. The answer, as we'll see later, is nothing — the p-value between them is 0 point 72.

Jane: And the baseline Hadamax control makes the comparison fair. It lands at 4 point 1 million parameters and 163 million FLOPs. Gamma-Hadamax-Valid reaches the same or better performance at 1 point 8 million parameters and 124 million FLOPs. The topology is doing real work.

Lalam: What strikes me is that this is effectively a hand-crafted architecture search guided by hypotheses — depth helps, early downsampling hurts, multiplicative interactions compound with depth. Each variant in the phase tests a specific structural claim.

Lu: And the parameter budgeting from earlier pays off here. Because they separate encoder parameters from head parameters, you can see exactly where bloat lives. For Eta, it's the head. For the Same variant, it's also the head. The encoder stays lean in both.

Meng: So page nineteen leaves us with two structurally similar architectures ready for training. They look nearly identical on paper, and that's precisely the point — tiny structural choices lead to measurable consequences.

Tom: And with the architectures set, the training protocol comes next — including a decision that I suspect raised a few eyebrows. They set weight decay to zero.

Training Without a Crutch: Tom: We're at the training configuration now, and page twenty-five contains the most provocative experimental choice in the entire paper. They set weight decay to exactly zero across every phase and every seed.

Jane: That's a direct break with PQN, which used ℓ2 regularization as part of its mathematical stability argument. The original analysis leans on it to bound the curvature of the nonlinear update and suppress value overestimation.

Lu: And the paper is completely upfront about the break. They devote an entire subsection to it, explaining that removing weight decay lets them isolate the contribution of architecture and normalization from the dampening effect of regularization.

Meng: But they also explicitly decline to claim a theoretical guarantee. No global bound on the TD Jacobian, no formal convergence proof. They say the combination of vectorized sampling, LayerNorm, and their architectural designs maintains operational stability across all tested seeds, and they leave it at that.

Tom: I find that more convincing than a hand-wavy proof would be. They removed the crutch, the system still walks, and here's the evidence across fifty-seven games and four seeds.

Jane: There's a practical angle too. Weight decay is a hyperparameter that somebody has to tune. Removing it removes a tuning knob, which fits this entire philosophy of simplifying the RL pipeline.

Lalam: And it sharpens the causal story. When your models beat the regularized PQN baseline by a large margin, you can't credit clever regularization tuning. The architecture is the explanation.

Lu: It also places the system right in the heart of the deadly triad — function approximation, bootstrapping, and off-policy data — which is famous for causing divergence. The discussion later argues that the gradual channel expansion and the max-pooling downsampling are what keep the system stable.

Meng: The rest of the training recipe is uniform across all experiments: RAdam optimizer, learning rate 2 point 5 times ten to the minus four, two hundred million frames, batch size 4096, 128 parallel environments. Every variant gets the same budget, and that comparability is what makes the architectural comparisons trustworthy.

Jane: One more detail from this section — the loss changes in Phase 3. The first two phases use mean squared error against the λ-return target, while Phase 3 switches to the HL-Gauss two-hot cross-entropy loss for the distributional heads.

Tom: So with training fixed and stability deliberately challenged, we finally hit the results. Phase 1 is all about the eight encoders, and the statistical machinery they bring to the comparison is genuinely impressive.

Phase 1 Results: Tom: Page thirty-one reports Phase 1, the encoder comparison. The headline is that Alpha, a four-layer variant, takes the top IQM at 3 point 536. Gamma, the five-layer variant, is right behind at 3 point 481, while the PQN baseline sits at 2 point 692.

Jane: Both beat the baseline by a large margin. But the paper doesn't crown Alpha, and the justification is statistical — the difference between Alpha and Gamma fails to reach significance under a Wilcoxon signed-rank test with Holm-Bonferroni correction.

Lu: There's also a robustness study. Ten random splits of the fifty-seven games into validation subsets of fifteen, and Gamma wins the efficiency tradeoff every single time. That's a deliberate defense against benchmark overfitting.

Meng: The failures are as informative as the winners. Eta, the shallow wide network with 23 point 8 million parameters, lands at 3 point 114 IQM. Raw capacity in the regression head simply cannot substitute for convolutional depth.

Tom: And Delta, with its giant 9-by-9 first-layer kernel, is significantly worse than the baseline — 2 point 374 IQM and a p-value under 0 point 001. Aggressive early spatial reduction destroys the fine-grained visual information that reactive control needs.

Jane: The probability of improvement matrix is the cleanest summary. Alpha, Beta, and Gamma all sit between 0 point 79 and 0 point 85 against PQN. Delta falls to 0 point 34, and Theta sits at exactly 0 point 50 — no better than a coin flip.

Lalam: That's the right way to read RL results. A single average can hide enormous variance. Pairwise probabilities across the full suite tell you about consistency, and consistency is what you want when choosing an architecture to build on.

Lu: The corrected significance matrices, with 36 pairwise comparisons and the family-wise error rate held at 0 point 05, give you the same message from a different angle. The ordering of the variants isn't noise.

Meng: So Phase 1 establishes the backbone: deeper is better when parameters are controlled, shallow width doesn't help, and giant kernels hurt. Gamma emerges as the balanced foundation, and that sets up Phase 2, where the Hadamax interactions come in.

Tom: And Phase 2 is where the performance really starts to jump. Let's take a look.

Phase 2 Results: Tom: Page thirty-seven opens Phase 2 with a complexity table, and the first thing that jumps out is Gamma-Hadamax-Valid holding the line at 1 point 84 million total parameters — nearly identical to plain Gamma — while the baseline Hadamax control swells to 4 point 1 million.

Jane: The performance gap is decisive. Gamma-Hadamax-Valid hits an IQM of 5 point 325, up from Gamma's 3 point 481, and the Wilcoxon test gives a p-value under 0 point 001. The multiplicative interaction is paying off on top of the deeper topology.

Lu: But the subtle result is the comparison between the two Gamma-Hadamax variants. Same, with the padding that doubles the head parameters to 3 point 5 million, performs no better. The p-value is 0 point 720. Extra capacity, zero measurable benefit.

Meng: The paper explains this through the receptive field analysis. Gamma covers a 39-by-39 patch of the input. The Hadamax variants expand that to 70-by-70

Page 8 of the paper: Tom: So we've just seen the IQM results for the final architecture, and now page 43 puts Aftab on a leaderboard with the big names in Atari.

Jane: That table is a reality check. Aftab's median score of 4 point 42 sits far above the classic DQN at 0 point 79, and it even beats Rainbow's 2 point 31, but it's well short of GDI's 11 point 46 and MuZero's 7 point 31.

Tom: The paper is careful not to overclaim there. They say straight out that those massive scores come from replay-buffer-dependent methods, and competing on raw performance isn't their goal.

Jane: Their goal is to show what a buffer-free agent can do when you actually design the encoder properly. And 4 point 42 median is more than four times human-level performance, which is a strong statement on its own.

Tom: What I like is the framing around memory and throughput. A replay buffer for Atari can eat gigabytes of RAM and create serious I/O bottlenecks during training. Aftab just doesn't have that problem.

Jane: Exactly. They're not saying they're the best agent ever. They're saying they're the best agent you can run without a large memory system, and that's a different trade-off space.

Tom: The caveat section is honest too. They admit the comparison isn't perfectly fair because other methods use different evaluation protocols and different compute budgets.

Jane: And they mention that a full Pareto analysis of wall-clock time, energy, and memory is left for future work. So they know this table is only part of the story.

Tom: But it's the part that makes the paper easy to summarize — a buffer-free agent that beats older replay-based methods and gets respectably close to modern ones.

Jane: So the natural next question is whether those results hold up when the environment itself changes, which is exactly what the Procgen runs test.

Page 9 of the paper: Tom: We've just seen the aggregate IQM scores, and now page 49 lays out the full per-game table that shows exactly where Aftab's gains come from.

Jane: That table is a wild ride. You've got Video Pinball at over a thousand human-normalized, Phoenix at 152, Atlantis at 143, and then you've got games like Demon Attack where Aftab scores just 6 point 9 compared to PQN's 72 point 5.

Tom: So the ensemble and distributional heads aren't universally better — they crush some games and stumble on others.

Jane: Right, and that's precisely why the IQM matters. It chops off the bottom quarter and the top quarter of the results, so those crazy highs and painful lows don't dominate the aggregate.

Tom: Looking at the median tells the same story. Aftab's median is 4 point 42, which is well above human level, but it's much lower than the IQM of 6 point 48 because the middle of the pack is more modest.

Jane: What's interesting is comparing the two finalists here. The Ensemble Dueling head alone gets a median of 3 point 65, and adding the distributional loss in Aftab pushes that to 4 point 42.

Tom: So the distributional piece isn't just a theoretical addition — it genuinely lifts the typical game performance, not just the outliers.

Jane: You also see some games where Aftab actually does worse than plain PQN, like Double Dunk and Skiing going negative. That's a reminder that this isn't a free win everywhere.

Tom: But the fact that the IQM and probability numbers hold up despite those failures is what makes the overall claim credible.

Jane: And that brings us to the next page, where the probability of improvement matrix tells us whether these wins are consistent across the suite or just a few lucky games.

Page 10 of the paper: Tom: We've just seen the full per-game table, and now page 55 shows the learning curves along with the opening of the statistical significance section.

Jane: That figure is actually really telling. It plots IQM across the entire 200 million frames, and then the zoomed panel on the right isolates the final 50 million frames.

Tom: The zoomed panel is where the stability story comes through. Aftab isn't just finishing higher — its line stays flat and high without those late-training dips that plague many RL agents.

Jane: That consistency suggests the ten head ensemble is doing real variance reduction, not just squeezing out a few lucky spikes.

Tom: And right after the figure, the text starts explaining the statistical machinery that backs up those curves.

Jane: They reintroduce the IQM and the Wilcoxon signed-rank test with the Holm-Bonferroni correction, but the new part is how explicitly they argue for filtering the data.

Tom: The IQM chops off the bottom and top quarters of the fifty-seven game scores, so one catastrophic failure or one enormous Video Pinball outlier can't drive the conclusion.

Jane: Exactly. And the pairwise tests across thirty-six comparisons include a strict correction for false positives. The page is saying the curves look good, and here's the formal proof that the differences aren't just noise.

Tom: That's a solid combination — visual evidence coupled with a rigorous statistical framework.

Jane: And with that foundation laid, the paper moves into its limitations, where they'll talk about what this architecture might not handle and where the theory still has gaps.

Conclusion: Tom: So we've walked through the whole Aftab paper, from the encoder benchmark through the Hadamax integration and the final composite architecture, and it's time to wrap up where this leaves the field.

Jane: The big picture is that the paper makes a strong case that the visual encoder has been a silent bottleneck in model-free RL for over a decade, and just replacing that piece can more than double your aggregate score.

Tom: What impressed me most is the discipline. They ran every variant with the same frame budget, the same seeds, the same hyperparameters, and then subjected every comparison to proper multiple-testing corrections.

Jane: And they were honest about the boundaries too. They never claim to beat the replay-buffer giants like GDI or MuZero, and they explicitly say their median score of 4 point 42 is not state-of-the-art.

Tom: Instead, they positioned Aftab as the strongest buffer-free option, which is a different and arguably more practical goal for people working under memory or throughput constraints.

Jane: The removal of weight decay is the boldest empirical claim. The original PQN leaned on that regularization as a mathematical crutch, and they showed the architecture alone can keep training stable across four seeds and fifty-seven games.

Tom: There are real limitations though. Everything is discrete-action, Atari-style, and the Procgen results are much more modest than the Atari gains.

Jane: The Procgen numbers actually make the paper more believable for me. If Aftab had crushed both benchmarks equally, I'd worry about some hidden leakage. Instead, you see a partial transfer, which is typical of RL systems.

Tom: The open-source release with model definitions and raw logs is a big deal for the community, because anyone can build on this without reimplementing from scratch.

Jane: And that's the natural bridge to our next paper, where the authors ask whether these architectural lessons carry over to continuous control, which is exactly the gap they left open.