2608.06963-Learning in Deep Networks under Dale’s Constraint

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Learning in Deep Networks under Dale’s Constraint".

Jane: The paper was written by Roy Abel and Shimon Ullman from Weizmann Institute of Science.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: I get to open this one, and I'm genuinely excited. This paper asks whether deep learning can work when every neuron obeys real biological rules — non-negative firing, fixed synaptic signs, local updates only. That's a much harder problem than it sounds.

Jane: Dale's law is the star of the show. A neuron is born excitatory or inhibitory and never changes identity. Most biologically plausible learning papers quietly ignore that, because it's awkward.

Lu: The authors build what they call an on-off motif. Two non-negative channels represent a signed value — one carries the positive part, the other the negative part. The sign lives in which channel fires.

Tom: The actual signal is the difference between the two channels. Then the learning rule is purely Hebbian — presynaptic activity times a feedback signal, all local. No global gradient anywhere in the loop.

Meng: Here's the part that got me: they prove this local rule recovers the exact backpropagation weight update. Not an approximation. Exact, under symmetric wiring.

Jane: Exact is a strong word, and I was skeptical at first. But the appendix walks through the whole induction, and the controlled experiments line up with the theory.

Lalam: Tiny ImageNet is what convinced me. Their constrained model reached 42.31 percent top-1 accuracy, and it beat vanilla convolutional networks with the same number of channels. The biologically honest model won.

Tom: So the claim that matters: the brain can keep Dale's law and still perform gradient-based credit assignment. The paired channels are a feature, not a workaround.

Jane: That flips the standard story. Biology stops being an obstacle and starts being part of the solution.

Meng: I still need to see the circuit though. How do two non-negative channels actually encode a minus sign?

Lu: And how does the error flow backward without a single signed neuron?

Tom: Hold those thoughts. The opening pages lay out why the problem is hard, and then we meet the motif — it's beautifully simple.

Page 1 of the paper: Tom: We've heard the punchline, so let's slow down and start at the very beginning. The opening pages give us the abstract and the introduction, and they're basically a list of why backprop can't be transplanted into the brain.

Jane: The usual suspects are all there — global objective functions, non-local credit assignment, closely matched forward and backward pathways. But the deepest obstacle is signed quantities.

Lu: A neuron can't fire at a negative rate. A synapse under Dale's law can't switch from exciting to inhibiting. So how do you transmit a minus sign through a network of plus-only neurons?

Tom: The authors walk through the existing toolbox. Predictive coding gets local errors. Equilibrium propagation gets local updates. Feedback alignment removes weight transport. Target propagation sends targets instead of gradients. None of them respect excitatory-inhibitory identity.

Jane: That's the gap they're pointing at. Error units in predictive coding are often signed — one unit can push both ways at its output. It still violates Dale.

Meng: The page hints at their answer: two complementary non-negative channels, like the ON and OFF pathways in the retina. And they cite Francioni's recent dendrite work — opposing positive and negative learning contributions arriving at cortical dendrites.

Lalam: So the opening is a promise of principle. They'll build a circuit motif, repeat it through the whole network, and prove it recovers backprop updates without ever sending a negative signal.

Lu: There's also an anatomical bet hidden here. The model uses the same motif in bottom-up and top-down streams, which mirrors real cortical structure.

Tom: And the abstract doesn't oversell — it claims substantial gains on Tiny ImageNet, which we've seen holds up. The problem is real, the prior work is real, and the proposed fix is concrete.

Jane: I like that they name the open question explicitly. How do you do supervised credit assignment with local mechanisms inside a Dale-constrained network? That's the exact gap.

Tom: Now we get to the interesting part — the actual circuit. Remember that seesaw question? The next pages answer it.

Page 2 of the paper: Jane: So here's the motif, and it's almost embarrassingly simple. Two inputs, two outputs, and a couple of interneurons in between. The whole trick fits in one small figure.

Tom: Each input excites its own output channel, and through an inhibitory interneuron it suppresses the opposite channel. That cross-inhibition is where the magic lives.

Lu: When the internal weights are all one, the circuit computes a difference. The On channel fires when the first input wins, the Off channel fires when the second wins.

Jane: With a threshold, both channels stay silent when the difference is tiny. So at most one channel is ever active — the sign is carried by which channel fires, not by any negative activity.

Meng: It's like a balance scale. You don't store the number minus three. You store three on the negative side.

Jane: Exactly. Every neuron still fires non-negative rates. The negativity lives in the channel identity, not in the firing rate.

Tom: Then the architecture steps up. Each hidden unit in a standard network becomes one of these motifs, so every scalar activity expands into a pair of channels.

Lu: Inter-layer connections are all excitatory, because the motif outputs are excitatory neurons. Negative influences come from the complementary channels, not from negative weights.

Meng: And they don't lock the internal weights to one. The motif can learn its internal magnitudes while preserving signs. That flexibility matters for real learning.

Tom: It also means the representation is honest — every connection in the whole network respects Dale's law. No hidden mixed-sign shortcuts.

Jane: The forward stream is settled, then. But the backward stream — the error signals — that's where the real gymnastics start. And that's what the paper tackles head-on.

Page 3 of the paper: Lu: Now the theory kicks in. The signed learning signal is split into two top-down populations — one that pushes synaptic weights up, one that pushes them down.

Tom: Their difference plays the role of the conventional backprop error. And the theorem says that difference propagates through the network exactly like backprop's delta.

Jane: The gating mechanism is the clever part. The top-down signal only flows through the channel that was active in the forward pass. That's the derivative of ReLU, implemented with real wiring.

Meng: So the proof is an induction. If the top layer represents the error correctly, and the feedback follows the symmetric weights, then every lower layer inherits the same recursion.

Tom: And the Hebbian rule then matches gradient descent: presynaptic activity times the difference of the two feedback channels. Clean, local, and exact.

Jane: There's an honest caveat — the theorem assumes aligned bottom-up and top-down weights. The authors test how much that alignment actually matters.

Lu: They run fully connected nets on MNIST, Fashion-MNIST, and CIFAR-10. Symmetric weights match backprop almost perfectly. Weakly aligned weights stay close.

Meng: But far-from-aligned weights collapse — on CIFAR-10 the accuracy drops from about 56.5 percent down to 41.7 percent. Alignment is doing real work.

Tom: They also test learned internal motifs. Sharing the motif weights within a layer gives the best numbers across all three datasets. That's a nice surprise.

Jane: So the theory says local learning equals backprop, and the controlled experiments say yes under symmetry, and robust to small deviations. Then the paper pauses, and the reference list tells you exactly which giants this work stands on.

Page 4 of the paper: Meng: The references open with Richards and colleagues — the argument that deep learning can serve as a computational framework for neuroscience. That's this paper's home turf.

Tom: Then the classics: Rumelhart's backprop, Hinton's forward-forward, Lillicrap's "Backpropagation and the brain." The paper is picking a fight with a very long shadow.

Jane: Predictive coding, equilibrium propagation, target propagation, dendritic microcircuits — the usual suspects all show up. You can see the authors mapping the whole landscape before they carve out their spot.

Lu: And the critique is visible in what they cite. Alonso and Neftci tightened non-negative firing rates in predictive coding, but critics argued that modifications reduced biological plausibility.

Tom: The Daleian network line is there too — Haber and Schneidman showed such networks can be expressive and robust, but their training still relied on backprop. That's exactly the gap this paper tries to fill.

Jane: The on-off inspiration comes straight from vision science. Kuffler's retina work, Schiller's ON/OFF channels, even simple-cell receptive field structure in visual cortex.

Meng: I like the Markov et al. citation — primate cortex has roughly twice as many top-down connections as bottom-up ones. That justifies their two parallel top-down networks.

Lu: So the references aren't decoration. Every citation marks a constraint the paper claims to satisfy better than its predecessors.

Tom: And one citation stands out — Francioni and colleagues, real dendrites carrying opposing instructive signals. The biology is moving toward this architecture.

Jane: The reference list tells you the paper's ambition: stitch together Hebbian learning, Dale's law, on-off vision, and backprop's credit assignment into one coherent story.

Meng: Now the appendices — that's where the real architecture lives, with all the dense wiring diagrams.

Page 5 of the paper: Lu: Right, the main text shows a simplified version. The appendix draws the full wiring, and it's dense. Every bottom-up motif gets two top-down motifs — one for its On channel, one for its Off channel.

Tom: So the feedback structure has four channels per forward unit. Two motifs, each with its own positive and negative update populations. That's the neural cost of staying Dale-compliant.

Jane: The gating comes first. Lateral connections from the bottom-up stream suppress the top-down pathway associated with the inactive channel. Only the active route gets feedback.

Meng: Then each top-down motif applies the same on-off difference computation. So feedback signals are also paired non-negative channels. No signed neurons anywhere in the loop.

Lu: The cross-channel connectivity is the subtle part. Signals can cross between the On and Off associations, and between the positive-update and negative-update populations. That crossing is what builds a signed error.

Tom: The result is three separate weight sets: bottom-up, positive-update feedback, and negative-update feedback. Each feedback matrix has the same shape as the forward one, just transposed.

Jane: The learning flow has three stages — forward pass, gated top-down pass, then local Hebbian updates everywhere. It reads like a recipe.

Meng: And they update the top-down weights with the same Hebbian logic, so alignment between pathways is preserved if it starts aligned. That's the engineering backbone of the theorem.

Lu: Without this wiring, the proof wouldn't have legs. The appendix makes the abstraction concrete.

Tom: Then come the training recipes — datasets, optimizers, hyperparameters. The practical side of the story.

Page 6 of the paper: Jane: The experimental appendix — the nitty-gritty. One NVIDIA A10 GPU. No enormous compute budget, which is refreshing.

Meng: The controlled runs use plain SGD, learning rate one times ten to the minus two, batch size 64, no weight decay, no schedule. Ten seeds, best epoch reported.

Lu: They define the alignment variants carefully. Symmetric is an exact copy of weights. Weak symmetric adds Gaussian noise with standard deviation 0.01. Noisy adds noise to the updates too. Asymmetric is fully random.

Tom: The asymmetric case is worth flagging — random high-dimensional vectors are nearly orthogonal, so those pathways are far apart. That's why performance tanks on CIFAR-10.

Jane: The motif variants are interesting. Learned Unit lets each unit tune its internal weights under sign constraints. Shared Learned Unit forces one internal set per layer.

Meng: Shared gives the best accuracy on every dataset. Sharing acts like a regularizer, apparently.

Lu: Then Tiny ImageNet uses five convolutional stages. The on-off actual channel counts double the vanilla numbers — six to one twenty-eight, then one twenty-eight to three eighty-four, and so on.

Tom: Training uses AdamW with learning rate 2.5 times ten to the minus four, batch size 256, 25 epochs, and a cosine schedule with warmup. Standard modern recipe.

Jane: The comparisons are careful too. One vanilla baseline matches effective units, the other matches the doubled channel count. The on-off model beats both.

Meng: So the improvement isn't just extra parameters. It's the paired representation itself. Now we get to the formal part — the proof, written out line by line.

Page 7 of the paper: Lu: Appendix four, the full proof. It starts with a simple decomposition — every signed vector splits into a positive part and a negative part. That's the whole idea in notation.

Tom: They define the top-down channels as the positive and negative parts of the descent signal. At the output layer, those channels are exact by construction.

Jane: Then the induction step. The positive channel propagates through its own feedback weights, the negative channel through its own weights. Both get gated by the bottom-up activity.

Meng: The gate matrix is the key — it's diagonal, with ones exactly where the ReLU was active. That's the derivative of ReLU, implemented by real circuitry.

Lu: Under symmetric connectivity, the two feedback weight matrices equal the transposed forward weights. Subtract the channels, and the recursion becomes backprop's recursion, exactly.

Tom: The algebra is clean. The difference of the propagated channels equals the propagated difference of the descent signal. Then induction finishes the job.

Jane: And the update rule? Presynaptic activity times the feedback difference. That's gradient descent, with a minus sign matching the loss derivative. The proof closes the loop.

Meng: The last part addresses separate bottom-up and top-down weights — if they're initialized close, matched Hebbian updates keep them close. That's the robustness story.

Lu: So the theorem isn't a flourish — it's the spine of the paper. Every experiment tests some assumption the proof relies on: symmetry, alignment, motif learning.

Tom: Then the paper wraps with the NeurIPS checklist. We should see what the authors are willing to claim about their own work.

Page 8 of the paper: Jane: The checklist jumps in with the first question: do the main claims match the contributions? The authors answer yes, without hedging.

Meng: Limitations — yes, they flag them at the end of the paper. The model still uses a lot of neurons, and it's far from a full cortical model.

Lu: Theory assumptions and proofs — yes, pointing to the appendix. They even number the theorem and provide the full proof, not just a sketch.

Tom: Reproducibility — yes. Full details in the appendix, plus code in the supplemental material. That's a concrete commitment.

Jane: Experimental setting — yes, data splits and hyperparameters are all specified. Statistical significance — yes, they report standard deviations over seeds.

Meng: I appreciate the compute disclosure — a single A10 GPU. That makes the results feel attainable for other labs.

Lu: The checklist is dry, but it tells you the authors know where the weak spots are. They preempt the reviewer questions.

Tom: They also declare broader impacts as not applicable — the work is foundational neuroscience, no deployment path. That's a fair call.

Jane: And they credit existing assets properly — datasets and prior work are all referenced. The answers continue on the next page, and they stay consistent.

Page 9 of the paper: Lu: The checklist continues. Broader impacts — still marked not applicable, because the implications are for neuroscience rather than deployed systems.

Tom: No crowdsourcing, no human subjects, no risks from released models. The answers are all not applicable, and they're justified.

Jane: The declaration about LLM usage — also not applicable. The core method development didn't involve language models.

Meng: Safeguards and licenses get clean answers — nothing high-risk is being released, and datasets are properly credited.

Lu: The final section confirms the research conforms to the code of ethics. No surprises, but it's good practice.

Tom: Honestly, these pages are the least glamorous part of the paper, but they matter. They make the work trustable.

Jane: And they show a research style — open, cautious, precise. The same style we saw in the experiments and the proof.

Meng: Now we're at the end. Let's pull back and say what the paper actually changes.

Conclusion: Tom: Time to wrap. The paper's core contribution is a circuit motif — two channels, cross-inhibition — that makes Dale's law and backprop coexist.

Jane: The theorem is the anchor. Local Hebbian learning with paired top-down signals reproduces the backprop update exactly, under symmetric wiring.

Meng: The experiments go further. The architecture beats vanilla networks on Tiny ImageNet, so the constraints improved learning rather than hurting it.

Lu: For neuroscience, it's a concrete hypothesis — on-off populations in cortex could be doing credit assignment exactly this way.

Tom: For machine learning, it suggests paired rectified channels are an efficient representation, not just a biological concession.

Lalam: The limitations are real — doubled neuron counts, an idealized wiring assumption, and still a long way from full cortical complexity. But this is a genuine step.

Jane: And the references point to the future. Dendritic evidence, top-down pathway counts, predictive coding — the pieces are aligning.

Tom: We'll keep an eye on follow-ups. Will someone test this circuit in a biological model? That's the obvious next question.

Meng: Or scale the approach to modern architectures? That's the engineering challenge.

Jane: For now, the paper gives us a real bridge between deep learning and the brain. A good place to stop.

Tom: Goodbye to this one. On to the next paper.