Learning in Deep Networks under Dale's Constraint

arXiv:2608.06963 · cs.AI · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Learning in Deep Networks under Dale’s Constraint".

Jane: The paper was written by Roy Abel and Shimon Ullman from Weizmann Institute of Science.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: I get to open this one, and I'm genuinely excited. This paper asks whether deep learning can work when every neuron obeys real biological rules — non-negative firing, fixed synaptic signs, local updates only. That's a much harder problem than it sounds.

Jane: Dale's law is the star of the show. A neuron is born excitatory or inhibitory and never changes identity. Most biologically plausible learning papers quietly ignore that, because it's awkward.

Lu: The authors build what they call an on-off motif. Two non-negative channels represent a signed value — one carries the positive part, the other the negative part. The sign lives in which channel fires.

Tom: The actual signal is the difference between the two channels. Then the learning rule is purely Hebbian — presynaptic activity times a feedback signal, all local. No global gradient anywhere in the loop.

Meng: Here's the part that got me: they prove this local rule recovers the exact backpropagation weight update. Not an approximation. Exact, under symmetric wiring.

Jane: Exact is a strong word, and I was skeptical at first. But the appendix walks through the whole induction, and the controlled experiments line up with the theory.

Lalam: Tiny ImageNet is what convinced me. Their constrained model reached 42.31 percent top-1 accuracy, and it beat vanilla convolutional networks with the same number of channels. The biologically honest model won.

Tom: So the claim that matters: the brain can keep Dale's law and still perform gradient-based credit assignment. The paired channels are a feature, not a workaround.

Jane: That flips the standard story. Biology stops being an obstacle and starts being part of the solution.

Meng: I still need to see the circuit though. How do two non-negative channels actually encode a minus sign?

Lu: And how does the error flow backward without a single signed neuron?

Tom: Hold those thoughts. The opening pages lay out why the problem is hard, and then we meet the motif — it's beautifully simple.

Page 1 of the paper: Tom: We've heard the punchline, so let's slow down and start at the very beginning. The opening pages give us the abstract and the introduction, and they're basically a list of why backprop can't be transplanted into the brain.

Jane: The usual suspects are all there — global objective functions, non-local credit assignment, closely matched forward and backward pathways. But the deepest obstacle is signed quantities.

Lu: A neuron can't fire at a negative rate. A synapse under Dale's law can't switch from exciting to inhibiting. So how do you transmit a minus sign through a network of plus-only neurons?

Tom: The authors walk through the existing toolbox. Predictive coding gets local errors. Equilibrium propagation gets local updates. Feedback alignment removes weight transport. Target propagation sends targets instead of gradients. None of them respect excitatory-inhibitory identity.

Jane: That's the gap they're pointing at. Error units in predictive coding are often signed — one unit can push both ways at its output. It still violates Dale.

Meng: The page hints at their answer: two complementary non-negative channels, like the ON and OFF pathways in the retina. And they cite Francioni's recent dendrite work — opposing positive and negative learning contributions arriving at cortical dendrites.

Lalam: So the opening is a promise of principle. They'll build a circuit motif, repeat it through the whole network, and prove it recovers backprop updates without ever sending a negative signal.

Lu: There's also an anatomical bet hidden here. The model uses the same motif in bottom-up and top-down streams, which mirrors real cortical structure.

Tom: And the abstract doesn't oversell — it claims substantial gains on Tiny ImageNet, which we've seen holds up. The problem is real, the prior work is real, and the proposed fix is concrete.

Jane: I like that they name the open question explicitly. How do you do supervised credit assignment with local mechanisms inside a Dale-constrained network? That's the exact gap.

Tom: Now we get to the interesting part — the actual circuit. Remember that seesaw question? The next pages answer it.

Page 2 of the paper: Jane: So here's the motif, and it's almost embarrassingly simple. Two inputs, two outputs, and a couple of interneurons in between. The whole trick fits in one small figure.

Tom: Each input excites its own output channel, and through an inhibitory interneuron it suppresses the opposite channel. That cross-inhibition is where the magic lives.

Lu: When the internal weights are all one, the circuit computes a difference. The On channel fires when the first input wins, the Off channel fires when the second wins.

Jane: With a threshold, both channels stay silent when the difference is tiny. So at most one channel is ever active — the sign is carried by which channel fires, not by any negative activity.

Meng: It's like a balance scale. You don't store the number minus three. You store three on the negative side.

Jane: Exactly. Every neuron still fires non-negative rates. The negativity lives in the channel identity, not in the firing rate.

Tom: Then the architecture steps up. Each hidden unit in a standard network becomes one of these motifs, so every scalar activity expands into a pair of channels.

Lu: Inter-layer connections are all excitatory, because the motif outputs are excitatory neurons. Negative influences come from the complementary channels, not from negative weights.

Meng: And they don't lock the internal weights to one. The motif can learn its internal magnitudes while preserving signs. That flexibility matters for real learning.

Tom: It also means the representation is honest — every connection in the whole network respects Dale's law. No hidden mixed-sign shortcuts.

Jane: The forward stream is settled, then. But the backward stream — the error signals — that's where the real gymnastics start. And that's what the paper tackles head-on.

Page 3 of the paper: Lu: Now the theory kicks in. The signed learning signal is split into two top-down populations — one that pushes synaptic weights up, one that pushes them down.

Tom: Their difference plays the role of the conventional backprop error. And the theorem says that difference propagates through the network exactly like backprop's delta.

Jane: The gating mechanism is the clever part. The top-down signal only flows through the channel that was active in the forward pass. That's the derivative of ReLU, implemented with real wiring.

Meng: So the proof is an induction. If the top layer represents the error correctly, and the feedback follows the symmetric weights, then every lower layer inherits the same recursion.

Tom: And the Hebbian rule then matches gradient descent: presynaptic activity times the difference of the two feedback channels. Clean, local, and exact.

Jane: There's an honest caveat — the theorem assumes aligned bottom-up and top-down weights. The authors test how much that alignment actually matters.

Lu: They run fully connected nets on MNIST, Fashion-MNIST, and CIFAR-10. Symmetric weights match backprop almost perfectly. Weakly aligned weights stay close.

Meng: But far-from-aligned weights collapse — on CIFAR-10 the accuracy drops from about 56.5 percent down to 41.7 percent. Alignment is doing real work.

Tom: They also test learned internal motifs. Sharing the motif weights within a layer gives the best numbers across all three datasets. That's a nice surprise.

Jane: So the theory says local learning equals backprop, and the controlled experiments say yes under symmetry, and robust to small deviations. Then the paper pauses, and the reference list tells you exactly which giants this work stands on.

Page 4 of the paper: Meng: The references open with Richards and colleagues — the argument that deep learning can serve as a computational framework for neuroscience. That's this paper's home turf.

Tom: Then the classics: Rumelhart's backprop, Hinton's forward-forward, Lillicrap's "Backpropagation and the brain." The paper is picking a fight with a very long shadow.

Jane: Predictive coding, equilibrium propagation, target propagation, dendritic microcircuits — the usual suspects all show up. You can see the authors mapping the whole landscape before they carve out their spot.

Lu: And the critique is visible in what they cite. Alonso and Neftci tightened non-negative firing rates in predictive coding, but critics argued that modifications reduced biological plausibility.

Tom: The Daleian network line is there too — Haber and Schneidman showed such networks can be expressive and robust, but their training still relied on backprop. That's exactly the gap this paper tries to fill.

Jane: The on-off inspiration comes straight from vision science. Kuffler's retina work, Schiller's ON/OFF channels, even simple-cell receptive field structure in visual cortex.

Meng: I like the Markov et al. citation — primate cortex has roughly twice as many top-down connections as bottom-up ones. That justifies their two parallel top-down networks.

Lu: So the references aren't decoration. Every citation marks a constraint the paper claims to satisfy better than its predecessors.

Tom: And one citation stands out — Francioni and colleagues, real dendrites carrying opposing instructive signals. The biology is moving toward this architecture.

Jane: The reference list tells you the paper's ambition: stitch together Hebbian learning, Dale's law, on-off vision, and backprop's credit assignment into one coherent story.

Meng: Now the appendices — that's where the real architecture lives, with all the dense wiring diagrams.

Page 5 of the paper: Lu: Right, the main text shows a simplified version. The appendix draws the full wiring, and it's dense. Every bottom-up motif gets two top-down motifs — one for its On channel, one for its Off channel.

Tom: So the feedback structure has four channels per forward unit. Two motifs, each with its own positive and negative update populations. That's the neural cost of staying Dale-compliant.

Jane: The gating comes first. Lateral connections from the bottom-up stream suppress the top-down pathway associated with the inactive channel. Only the active route gets feedback.

Meng: Then each top-down motif applies the same on-off difference computation. So feedback signals are also paired non-negative channels. No signed neurons anywhere in the loop.

Lu: The cross-channel connectivity is the subtle part. Signals can cross between the On and Off associations, and between the positive-update and negative-update populations. That crossing is what builds a signed error.

Tom: The result is three separate weight sets: bottom-up, positive-update feedback, and negative-update feedback. Each feedback matrix has the same shape as the forward one, just transposed.

Jane: The learning flow has three stages — forward pass, gated top-down pass, then local Hebbian updates everywhere. It reads like a recipe.

Meng: And they update the top-down weights with the same Hebbian logic, so alignment between pathways is preserved if it starts aligned. That's the engineering backbone of the theorem.

Lu: Without this wiring, the proof wouldn't have legs. The appendix makes the abstraction concrete.

Tom: Then come the training recipes — datasets, optimizers, hyperparameters. The practical side of the story.

Page 6 of the paper: Jane: The experimental appendix — the nitty-gritty. One NVIDIA A10 GPU. No enormous compute budget, which is refreshing.

Meng: The controlled runs use plain SGD, learning rate one times ten to the minus two, batch size 64, no weight decay, no schedule. Ten seeds, best epoch reported.

Lu: They define the alignment variants carefully. Symmetric is an exact copy of weights. Weak symmetric adds Gaussian noise with standard deviation 0.01. Noisy adds noise to the updates too. Asymmetric is fully random.

Tom: The asymmetric case is worth flagging — random high-dimensional vectors are nearly orthogonal, so those pathways are far apart. That's why performance tanks on CIFAR-10.

Jane: The motif variants are interesting. Learned Unit lets each unit tune its internal weights under sign constraints. Shared Learned Unit forces one internal set per layer.

Meng: Shared gives the best accuracy on every dataset. Sharing acts like a regularizer, apparently.

Lu: Then Tiny ImageNet uses five convolutional stages. The on-off actual channel counts double the vanilla numbers — six to one twenty-eight, then one twenty-eight to three eighty-four, and so on.

Tom: Training uses AdamW with learning rate 2.5 times ten to the minus four, batch size 256, 25 epochs, and a cosine schedule with warmup. Standard modern recipe.

Jane: The comparisons are careful too. One vanilla baseline matches effective units, the other matches the doubled channel count. The on-off model beats both.

Meng: So the improvement isn't just extra parameters. It's the paired representation itself. Now we get to the formal part — the proof, written out line by line.

Page 7 of the paper: Lu: Appendix four, the full proof. It starts with a simple decomposition — every signed vector splits into a positive part and a negative part. That's the whole idea in notation.

Tom: They define the top-down channels as the positive and negative parts of the descent signal. At the output layer, those channels are exact by construction.

Jane: Then the induction step. The positive channel propagates through its own feedback weights, the negative channel through its own weights. Both get gated by the bottom-up activity.

Meng: The gate matrix is the key — it's diagonal, with ones exactly where the ReLU was active. That's the derivative of ReLU, implemented by real circuitry.

Lu: Under symmetric connectivity, the two feedback weight matrices equal the transposed forward weights. Subtract the channels, and the recursion becomes backprop's recursion, exactly.

Tom: The algebra is clean. The difference of the propagated channels equals the propagated difference of the descent signal. Then induction finishes the job.

Jane: And the update rule? Presynaptic activity times the feedback difference. That's gradient descent, with a minus sign matching the loss derivative. The proof closes the loop.

Meng: The last part addresses separate bottom-up and top-down weights — if they're initialized close, matched Hebbian updates keep them close. That's the robustness story.

Lu: So the theorem isn't a flourish — it's the spine of the paper. Every experiment tests some assumption the proof relies on: symmetry, alignment, motif learning.

Tom: Then the paper wraps with the NeurIPS checklist. We should see what the authors are willing to claim about their own work.

Page 8 of the paper: Jane: The checklist jumps in with the first question: do the main claims match the contributions? The authors answer yes, without hedging.

Meng: Limitations — yes, they flag them at the end of the paper. The model still uses a lot of neurons, and it's far from a full cortical model.

Lu: Theory assumptions and proofs — yes, pointing to the appendix. They even number the theorem and provide the full proof, not just a sketch.

Tom: Reproducibility — yes. Full details in the appendix, plus code in the supplemental material. That's a concrete commitment.

Jane: Experimental setting — yes, data splits and hyperparameters are all specified. Statistical significance — yes, they report standard deviations over seeds.

Meng: I appreciate the compute disclosure — a single A10 GPU. That makes the results feel attainable for other labs.

Lu: The checklist is dry, but it tells you the authors know where the weak spots are. They preempt the reviewer questions.

Tom: They also declare broader impacts as not applicable — the work is foundational neuroscience, no deployment path. That's a fair call.

Jane: And they credit existing assets properly — datasets and prior work are all referenced. The answers continue on the next page, and they stay consistent.

Page 9 of the paper: Lu: The checklist continues. Broader impacts — still marked not applicable, because the implications are for neuroscience rather than deployed systems.

Tom: No crowdsourcing, no human subjects, no risks from released models. The answers are all not applicable, and they're justified.

Jane: The declaration about LLM usage — also not applicable. The core method development didn't involve language models.

Meng: Safeguards and licenses get clean answers — nothing high-risk is being released, and datasets are properly credited.

Lu: The final section confirms the research conforms to the code of ethics. No surprises, but it's good practice.

Tom: Honestly, these pages are the least glamorous part of the paper, but they matter. They make the work trustable.

Jane: And they show a research style — open, cautious, precise. The same style we saw in the experiments and the proof.

Meng: Now we're at the end. Let's pull back and say what the paper actually changes.

Conclusion: Tom: Time to wrap. The paper's core contribution is a circuit motif — two channels, cross-inhibition — that makes Dale's law and backprop coexist.

Jane: The theorem is the anchor. Local Hebbian learning with paired top-down signals reproduces the backprop update exactly, under symmetric wiring.

Meng: The experiments go further. The architecture beats vanilla networks on Tiny ImageNet, so the constraints improved learning rather than hurting it.

Lu: For neuroscience, it's a concrete hypothesis — on-off populations in cortex could be doing credit assignment exactly this way.

Tom: For machine learning, it suggests paired rectified channels are an efficient representation, not just a biological concession.

Lalam: The limitations are real — doubled neuron counts, an idealized wiring assumption, and still a long way from full cortical complexity. But this is a genuine step.

Jane: And the references point to the future. Dendritic evidence, top-down pathway counts, predictive coding — the pieces are aligning.

Tom: We'll keep an eye on follow-ups. Will someone test this circuit in a biological model? That's the obvious next question.

Meng: Or scale the approach to modern architectures? That's the engineering challenge.

Jane: For now, the paper gives us a real bridge between deep learning and the brain. A good place to stop.

Tom: Goodbye to this one. On to the next paper.

Roy Abel, Shimon Ullman

Weizmann Institute of Science

cs.AI

Submitted: 2026-08-07

Updated: 2026-08-10

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: The paper is authored by Roy Abel and Shimon Ullman from the Weizmann Institute of Science (arXiv:2608.06963v1, 2026).

Key concepts

Dale's law: A biological rule stating that a neuron is born either excitatory or inhibitory and never changes identity, so its synapses keep fixed signs. This makes transmitting negative signals difficult, and many biologically plausible learning models ignore it. The paper treats it as a core constraint to satisfy.
On-off motif: A circuit motif with two non-negative channels representing a signed value: one carries the positive part and the other the negative part. The actual signal is the difference between the channels, and the sign is carried by which channel fires. This lets a network transmit negative values using only non-negative firing rates.
Local Hebbian learning: A learning rule that updates synaptic weights using only local information: presynaptic activity times a feedback signal. The paper proves that, under symmetric wiring, this local rule reproduces the exact weight update of backpropagation, so gradient-based credit assignment can work without a global error signal.
Feedback alignment: The paper's theorem relies on top-down feedback weights being aligned with bottom-up forward weights. With exact symmetric weights, the local rule matches backprop exactly; with weakly aligned weights it stays close. Random, far-from-aligned weights cause performance to collapse, showing alignment is essential.

Terminology

Alignment — 22× in this paper · explained in 1 episode
Credit assignment — 13× in this paper · explained in 3 episodes
Benchmark — 11× in this paper · explained in 2 episodes
Predictive Coding — 9× in this paper · explained in 2 episodes
Core — 8× in this paper · explained in 2 episodes
Accuracy — 7× in this paper · explained in 1 episode
Adam — 2× in this paper · explained in 2 episodes
Institutional Review Board — 2× in this paper
Regularization — 2× in this paper · explained in 3 episodes
Robustness — 2× in this paper · explained in 5 episodes
Activation Function — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 3 episodes
Computational Biology — 1× in this paper · explained in 2 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Objective Function — 1× in this paper · explained in 2 episodes
Sample Efficiency — 1× in this paper · explained in 4 episodes

Summary

The paper is authored by Roy Abel and Shimon Ullman from the Weizmann Institute of Science (arXiv:2608.06963v1, 2026).

The paper addresses a fundamental discrepancy between biologically plausible learning models and a basic property of cortical circuitry known as Dale's constraint. As stated in the abstract:

"Biologically plausible learning models aim to explain how neural circuits can implement effective learning under the constraints of real neurons. Although significant progress has been made, a major remaining challenge is that existing models often allow neurons or synapses to represent mixed-sign values, both positive and negative, in violation of a basic aspect of cortical circuitry – Dale’s constraint: biological neurons are either excitatory or inhibitory, but not both, and synapses cannot change sign."

The authors note that artificial networks rely on signed quantities (activations, errors, gradients, weights), whereas biological circuits have non-negative firing rates and sign-constrained synapses. A biologically plausible learning mechanism must therefore address how to compute credit assignment locally and how to do so in a Dale-constrained network using only non-negative neural activity and fixed-sign connectivity.

The central contribution is an on-off neural circuit motif:

Our approach uses two complementary interacting non-negative channels to represent positive and negative contributions, inspired by evidence of on-off representations in the brain.

The architecture satisfies three key Daleian properties:

"(i) it represents signed quantities using only non-negative neuronal activity, (ii) all synapses have a fixed sign synaptic influence, and (iii) all synapses from a cell to all its target neurons are of the same sign, excitatory or inhibitory."

Motif computation: The basic motif receives two non-negative inputs, x1, x2 ≥ 0, and produces two non-negative output channels y+ and y−. The circuit contains excitatory input/output neurons and intermediate inhibitory neurons mediating cross-channel inhibition. With unit-magnitude internal weights, the circuit computes the signed contrast d = x1 − x2 via complementary rectifying nonlinearities:

y+ = ReLU+ θ(d):= max(d − θ, 0), y− = ReLU− θ(d):= max(−d − θ, 0)

Thus, a signed value is encoded through channel identity—positive differences propagate through the On channel, negative differences through the Off channel—while at most one channel is active.

Network construction: Full networks are built by replacing each hidden mixed-sign neuron with an on-off motif, expanding each scalar activity into a pair of non-negative channels. Inter-layer connections are excitatory, with negative influences mediated by complementary channels and intra-motif inhibitory circuitry. The same motif is used in both bottom-up (BU) and top-down (TD) streams.

Top-down propagation: For each BU motif, two corresponding TD motifs are associated (one per BU channel). TD propagation combines three operations involving lateral BU-TD gating (analogous to the derivative of a rectifying nonlinearity), intra-layer on-off processing, and inter-layer TD connectivity to lower layers.

Local Hebbian learning rule: Synaptic updates combine presynaptic BU activity with paired TD feedback signals:

∆Wiⱼ = ηxⱼ ∂̄i (positive TD channel drives increase)

∆Wiⱼ = −ηxⱼ ∂i (negative TD channel drives decrease)

Combined, the effective update is:

∆Wiⱼ = ηxⱼ(∂̄i − ∂i)

The effective signed feedback signal is ∂i = ∂̄i − ∂i, determined by the relative activity of two separated TD populations rather than a signed error neuron. The rule is local, and TD synapses receive matched updates ensuring BU-TD alignment is preserved.

The paper proves that the local learning scheme can exactly recover backpropagation updates:

Theorem. "Assume that (i) the BU network consists of linear operations, such as fully connected, locally connected, convolutional layers, or residual connections, followed by ReLU nonlinearities, and (ii) the BU and TD weights are aligned in their weights. Let δl denote the standard backpropagation error signal at layer l. Then the effective TD signal satisfies the same recursion as δl: ∂̄l − ∂l = δl. Consequently, the on-off Hebbian learning rule yields the same weight update as backpropagation."

The proof shows that TD propagation satisfies ∂̄l−1 = Gate hl−1 WlT ∂̄l and ∂l−1 = Gate hl−1 WlT ∂l. Taking differences under symmetric channel connectivity gives:

∂̄l−1 − ∂l−1 = σ′(hl−1)WlT(δl) = δl−1

Since the gating operation corresponds to the ReLU derivative, the recursion matches backpropagation exactly. When BU and TD weights are parameterized separately, matched local updates preserve alignment; if initialized close to alignment, the TD computation approximates backpropagation.

Controlled experiments (MNIST, Fashion-MNIST, CIFAR-10): The paper evaluates fully connected two-layer networks with 250 hidden on-off units (MNIST/Fashion-MNIST) and 500 units (CIFAR-10), batch size 64, learning rate 10−2. Key findings from Table 1:

On-Off Sym matches BP on the same architecture: 97.796% vs 97.792% (MNIST), 88.147% vs 88.100% (Fashion-MNIST), 56.483% vs 56.432% (CIFAR-10).
On CIFAR-10, on-off Sym (56.483%) outperforms both vanilla MLP (54.057%) and doubled-width MLP (55.510%).
Weak Sym and Noisy variants (approximate alignment, noisy updates) remain close to the symmetric reference, while far-from-alignment (Asym) initialization harms performance (41.707% on CIFAR-10).
The Shared Learned Unit variant (learnable internal motif weights shared within a layer) gives the best results on all datasets (98.027% MNIST, 88.276% Fashion-MNIST, 56.484% CIFAR-10), suggesting a useful regularization effect.

Tiny ImageNet scaling: Following the SCFF experimental setting with a five-convolution-layer architecture, results from Table 2 show:

"on-off Sym reaches 42.31% top-1 and 67.70% top-5 accuracy, outperforming SCFF by a large margin. It also outperforms both vanilla convolutional baselines: the matched-effective-width model reaches 35.58% top-1, and the matched-neuron-count ×2 model reaches 37.10%."

The authors emphasize this improvement cannot be explained by the doubled-channel representation alone: the on-off model outperforms vanilla networks even with equal neuron/channel counts, suggesting the paired-channel architecture provides an efficient representation relative to vanilla networks of comparable size. Visualization of learned activations (Appendix A.3) shows complementary on-off channels learning boundary-sensitive and color/contrast-sensitive representations—opposite edge polarities and complementary foreground/background structure—rather than redundant duplicated features.

The authors interpret the model in light of biological evidence:

"two TD populations play opposing instructive roles, with one driving Hebbian increases and the other driving decreases of synaptic weights. This is in line with biological observations of on-off and opponent channels in sensory systems... as well as in higher-level, e.g., in face and expression perception."

They relate the work to theoretical proposals that feedback may induce error-like differences between neural activities (Lillicrap et al., 2020) and recent evidence for vectorized instructive signals in cortical dendrites (Francioni et al., 2026).

The authors acknowledge two main limitations:

The current model focuses on one important component, but it is still far from providing a full biologically plausible model for cortical learning, and significant research is still required.

"the on-off model requires a substantial increase in the number of neurons, compared with models that use a single top-down network rather than two parallel networks used by the current model. However, it is consistent with the experimental estimate that the number of top-down cortical connections in primates is about twice that of bottom-up ones [Markov et al., 2014] and, as shown by the model, the two-network architecture can increase the model’s performance."

The paper demonstrates that effective learning can emerge from biologically plausible mechanisms without requiring mixed-sign signals, providing a step toward more realistic models of neural computation. The on-off architecture provides a concrete mechanism for signed credit assignment under non-negative activity and Dale-constrained connectivity, is theoretically grounded in exact backpropagation recovery, and empirically outperforms comparable vanilla networks on challenging benchmarks.

Improvements for AI systems

Based on this paper, AI systems can be improved in the following ways:

Dale-constrained neural architectures: Build networks with two complementary non-negative channels (on/off) per neuron, replacing signed activations and sign-changing weights. All synapses have fixed sign (excitatory or inhibitory). This allows AI models to run on neuromorphic hardware that enforces Dale's constraint, while still representing signed values through channel identity.
Local, backprop-equivalent learning without global error signals: Use paired top-down feedback channels that drive Hebbian increases and decreases of synaptic weights. The difference between the two feedback channels computes the exact backpropagation error signal via a local recursion. This enables training deep networks with updates that are mathematically equivalent to backpropagation but require only local presynaptic activity and paired TD signals—no global gradient propagation, no signed error neurons, and no weight transport.
Complementary feature learning: The on-off motif produces complementary rectified outputs (ReLU on positive and negative contrasts), causing channels to learn opposite edge polarities, boundary-sensitive versus color/contrast-sensitive features, and foreground/background structure. AI systems gain a richer, more efficient representation than vanilla networks of the same width, improving accuracy without increasing neuron count.
Matched feedforward-feedback alignment: By maintaining aligned bottom-up and top-down weights through matched local updates, the approximate version of the algorithm remains close to backprop performance. This allows AI systems to train with separate forward and feedback pathways—useful for online learning, temporal credit assignment, and modular/brain-inspired architectures.
Regularization via shared learned motif internals: The Shared Learned Unit variant—where internal motif weights are learnable but shared within a layer—provided the best accuracy on all tested datasets (98.027% on MNIST, 88.276% on Fashion-MNIST, 56.484% on CIFAR-10). AI systems can adopt this tied-weight motif design as a regularizer to improve generalization.
Superior scaling on large benchmarks: On Tiny ImageNet with a 5-layer convolutional architecture, the on-off network reached 42.31% top-1 accuracy versus 35.58% for an equal-width vanilla baseline and 37.10% for a doubled-neuron vanilla baseline. This demonstrates that the paired-channel representation provides real efficiency gains beyond parameter count—AI systems can achieve higher accuracy at fixed model size on challenging visual tasks.

Train deep supervised networks to near-backprop accuracy under strict biological constraints (non-negative activity, fixed-sign synapses, local plasticity rules).
Deploy on energy-efficient, neuromorphic, or mixed-sign hardware where negative activations and dynamic weight signs are physically impossible.
Learn richer complementary feature representations automatically, improving accuracy on image classification (MNIST, Fashion-MNIST, CIFAR-10, Tiny ImageNet) compared to standard networks of equal size.
Perform online/local learning without a central error module, enabling distributed, parallel, and possibly continual learning in edge devices.
Maintain high accuracy even with noisy, misaligned, or approximately aligned feedback connections, making the system robust to non-ideal hardware implementations.

Abstract

Biologically plausible learning models aim to explain how neural circuits can implement effective learning under the constraints of real neurons. Although significant progress has been made, a major remaining challenge is that existing models often allow neurons or synapses to represent mixed-sign values, both positive and negative, in violation of a basic aspect of cortical circuitry -- Dale's constraint: biological neurons are either excitatory or inhibitory, but not both, and synapses cannot change sign. In this work, we address this discrepancy by introducing a biologically motivated neural architecture in which both neural activations and learning signals are represented by non-negative activity, and synapses have fixed sign, while still supporting backpropagation-like learning. Our approach uses two complementary interacting non-negative channels to represent positive and negative contributions, inspired by evidence of on-off representations in the brain. These channels are implemented through a simple neural circuit motif, which is repeated throughout the network in both bottom-up and top-down pathways. Combined with a local Hebbian learning rule, the resulting model propagates learning signals and updates weights using only local interactions between neurons. We show theoretically that our learning scheme can exactly recover the backpropagation update despite relying solely on non-negative error signals. Empirically, beyond satisfying stronger biological constraints, the on-off architecture learns efficient representations, yielding substantial gains over comparable vanilla networks on the Tiny ImageNet benchmark. These results demonstrate that effective learning can emerge from biologically plausible mechanisms without requiring mixed-sign signals, providing a step toward more realistic models of neural computation.