Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks

summary

Video file (mp4)

The gist

The gist The pairing of bidirectional PC (bPC) and error-based reparameterization (ePC) enables deep, 4-bit predictive coding networks that retain all canonical properties at depths up to eight

In short

The paper introduces ePC+bPC, combining bidirectional predictive coding and error-based reparameterization to create deep predictive coding networks that work well at 4-bit precision. This method overcomes signal decay and error instability issues found in standard models, allowing networks to train up to eight layers while maintaining high accuracy and generation capabilities under quantization.

Key concepts

Predictive Coding Networks (PCNs)
A type of hierarchical learning algorithm where each layer tries to predict the activity of the layer above it. They are unique because they can learn from local information, generate data, and use energy minimization for inference, properties that standard backpropagation networks lack.
Bidirectional PC (bPC)
This approach unifies discrimination (predicting input) and generation (producing output). It involves using both top-down generative weights and bottom-up discriminative weights simultaneously. This structure is key to achieving the full set of canonical PCN properties in deep networks.
Error-Based Reparameterization (ePC)
Instead of optimizing neural states during inference, ePC optimizes prediction errors. This allows the output signal to reach every layer in a single backward pass, eliminating the exponential decay problem common in state-based models. This method is crucial for deep inference.
Global-Norm Clipping
This technique stabilizes error dynamics by clipping the norm of error gradients across all layers during training. It prevents the errors from growing unboundedly as depth increases, effectively stopping the divergence that occurs when optimizing prediction errors without proper regularization.

Terminology used across episodes

This episode discusses

The paper

Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks · Read on arXiv

Garrett Kinman, Željko Žilic

McGill University

Predictive coding networks (PCNs) promise properties not offered by backpropagation-trained networks: learning from purely local information, a single network that both discriminates and generates, and flexible inference by energy minimization. However, their iterative inference is expensive, making hardware acceleration desirable, and a useful, efficient accelerator needs both depth and aggressive quantization. Canonical state-based PC delivers neither: its inference signal decays exponentially with depth, so deep models fail to train, and low precision makes it worse. We show that pairing two recent methods carries the full PCN property set into the deep, 4-bit regime: bidirectional PC (bPC), which unifies discrimination and generation, and the error-based reparameterization (ePC), which optimizes prediction errors rather than states during inference. Error optimization delivers the output signal to every layer in a single backward pass, removing the signal decay; one additional element, global-norm clipping of the error gradients, stabilizes the error dynamics at depth. The resulting ePC+bPC model trains stably up to at least eight layers and retains its full-precision accuracy and clean class-conditional generation under W4A4E4 quantization-aware training, across FP4 and three posit formats. On six-layer networks, where state-based bPC collapses to chance, ePC+bPC classifies more than 50 accuracy points above it after one epoch of training, with class-conditional generations 5-9x better in RMSE. 4-bit quantization is nearly free: full-precision and 4-bit accuracies differ by at most 5 points, and by under two in the best-performing formats. We characterize the two failure modes the method overcomes, bounded signal decay in state-based PC and error divergence in unregularized error optimization, and discuss implications for a 4-bit-native PCN accelerator.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks".

Jane: The gist The pairing of bidirectional PC (bPC) and error-based reparameterization (ePC) enables deep, 4-bit predictive coding networks that retain all canonical properties at depths up to eight layers.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: The paper proposes a specific pairing of two methods to carry the full set of predictive coding network properties into the deep four-bit regime. They introduce bidirectional predictive coding, or bPC, which unifies discrimination and generation, alongside error-based reparameterization, or ePC.

Jane: What they claim is that this combination delivers those three properties—local learning, native generation, and flexible energy-based inference—even at depths up to eight layers when using four bits for quantization.

Lu: The core idea they push is to optimize prediction errors instead of the neural states during inference. They set up a network with top-down generative weights W and bottom-up discriminative weights V.

Meng: So how does optimizing the error change things? Can we just tweak the training loop to do that?

Tom: Exactly. In training, they adjust stored per-layer prediction errors instead of states. This lets them deliver the output loss signal to every layer in a single global backpropagation pass, which removes that signal decay.

Jane: And they add one more thing for stability: global-norm clipping of the error gradients at depth to keep the error dynamics from diverging. That's what makes their ePC+bPC model train stably up to at least eight layers.

Lalam: They show this works with W4A4E4 quantization-aware training across FP4 and three posit formats, maintaining full-precision accuracy and clean class-conditional generation under those settings.

Lu: What’s really interesting is how they handle the failure modes of the other methods. State-based bPC fails by bounded signal decay where the inference signal shrinks geometrically, while unregularized error optimization fails by error divergence because errors are directly optimized without bound.

Meng: So, what’s the trade-off there? They're adding a backward pass per inference step, which might sacrifice strict neuromorphic locality but lets the output signal reach every layer at once.

Conclusion: Tom: Looking at this paper, "Error-Based Bidirectional Predictive Coding Enables Deep, four-Bit Predictive Coding Networks," what really stands out is how they solve the core issue of signal decay in deep predictive coding models when you push them into low precision <ref:2610.15933#pg1,Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks>.

Jane: They show that pairing bidirectional PC with error-based reparameterization is the way to get those canonical properties alive at depths of eight layers in four bits.

Lu: It moves the focus from optimizing states to optimizing prediction errors, which fundamentally changes how you train and infer these hierarchical networks.

Meng: For someone thinking about deployment, this suggests that a four-bit native datapath is viable for deep predictive coding because the accuracy drop is only about five points compared to full precision at six layers <ref:2610.15933#pg1>.

Lalam: The authors clarify that the problems they fixed aren't actually quantization of errors to zero, but rather bounded signal decay in state-based PC and error divergence in unregularized error optimization.

Tom: So, the implication here is that for hardware design targeting a four-bit native predictive coding network, you don't have to worry about the inference signal vanishing exponentially with depth anymore if you use this approach <ref:2610.15933#pg1>.

Jane: It gives us a path toward building deep predictive coding accelerators that actually work efficiently on low precision data.

More episodes

← Home