An all-transistor physical-learning transformer can be trained to perform in-context learning

summary

The gist

The gist An all-transistor physical-learning transformer can be trained to perform in-context learning by explicitly restoring bidirectional information flow in a double or dyadic circuit to enable

In short

The paper shows how a physical circuit, modeled using MOSFETs, can be trained to act as a transformer for in-context learning tasks. It achieves this by explicitly restoring bidirectional information flow in a double circuit, which allows training with the necessary cross-entropy loss. This enables the system to learn associative recall tasks that are hard for standard neural networks.

Key concepts

Physical Learning Paradigm: This is a machine learning method where an analog physical system handles both the forward calculation and back-propagation of errors. The circuit settles into a steady state, and error signals are injected to find a new state that minimizes energy (power dissipation). This allows for precise gradient reconstruction.
Restoring Bidirectional Flow: Ideal circuits often block error signals from flowing backward. To fix this, the authors add extra elements to a double or dyadic circuit. This setup uses the mean of node voltages for forward computation and the difference for the error signal, mimicking how automatic differentiation works in complex systems.
Cross-Entropy Loss Training: Standard training often uses mean squared error, but this system requires cross-entropy loss to succeed in learning. The authors create a novel floating circuit element based on the MOSFET sub-circuit used for softmax to generate the specific nudges needed for this loss function during training.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Flow Matching — explained in 16 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes

This episode discusses

The paper

An all-transistor physical-learning transformer can be trained to perform in-context learning · Read on arXiv

Michael C. Abbott, Benjamin B. Machta

Yale Physics & QBio

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "An all-transistor physical-learning transformer can be trained to perform in-context learning".

Jane: The gist An all-transistor physical-learning transformer can be trained to perform in-context learning by explicitly restoring bidirectional information flow in a double or dyadic circuit to enable training with cross-entropy…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So let’s talk more about this paper, "An all-transistor physical-learning transformer can be trained to perform in-context learning." We already touched on how they claim that standard physical learning systems struggle with associative tasks and how these struggle compared to transformers > Page one - <ref:2610.12571#pg3>.

Jane: Right. The main thesis seems to be that they are going beyond just equilibrium propagation by explicitly restoring the bidirectional information flow in a double or dyadic circuit, which is necessary because ideal voltage multipliers are non-reciprocal > Page two - <ref:2610.12571#pg3>.

Lu: They argue that this restoration allows them to construct an attention mechanism by exploiting the exponential current law of subthreshold MOSFET devices, which is a natural way to replace the dot product with something more analogous to Bahdanau attention > Page two - <ref:2610.12571#pg3>.

Meng: So they're taking an MLP structure and making it work as a transformer by fixing the multiplication problem through this circuit modification > Page two - <ref:2610.12571#pg3>. I wonder if that’s feasible when you have to deal with these very specific transistor dynamics.

Lalam: Lalam confirms that in simulations, this circuit can be trained to perform a simple in-context learning task, even though it uses realistic reverse components which only transmit an approximate gradient > Page one - <ref:2610.12571#pg3>. It shows the system can learn something meaningful despite the imperfections.

Tom: That’s cool because it means we don't need perfect physics just to start training on these complex tasks; we just need a nudge in the right direction > Page one - <ref:2610.12571#pg3>. But they stress that this specific training requires cross-entropy loss, not mean squared error > Page one - <ref:2610.12571#pg3>.

Jane: So, if you’re listening who doesn’t read this paper, what does it really mean? It means AI can learn tasks that require associative recall and context in a way traditional neural networks couldn't achieve > Page one - <ref:2610.12571#pg3>.

Lu: It suggests that the architecture itself has potential when implemented in an analog physical system, but you just have to build the right circuit structure to unlock that potential > Page two - <ref:2610.12571#pg3>. It’s not just about brute force computation.

Meng: From a production side, I have to wonder about the cost of building those custom circuit elements—they sound like they add complexity that might make them too expensive for real-world deployment > Page one - <ref:2610.12571#pg3>. What breaks at scale is often the bespoke parts.

Lalam: Lalam has seen the literature, and they connect this to how it changes what we need in terms of hardware design for AI > Page two - <ref:2610.12571#pg3>. It’s about designing circuits that inherently support learning rather than just computation > Page two - <ref:2610.12571#pg3>.

Jane: So, if you’re listening who doesn’t read this paper, the big idea is that analog systems can be trained to do complex things without needing perfect physics all the time > Page one - <ref:2610.12571#pg3>.

Conclusion: Tom: So to wrap up, this paper on "An all-transistor physical-learning transformer can be trained to perform in-context learning" shows that by fixing the flow issue in a double or dyadic circuit, you can build a transformer architecture that learns associative recall tasks > Page two - <ref:2610.12571#pg3>.

Jane: And the authors are Michael C. Abbott and Benjamin B. Machta, and what this means is that we might see physical systems move from just being computation engines to being learning agents capable of understanding context > Page one - <ref:2610.12571#pg3>.

Lu: It pushes the idea that the architecture matters more than just the material it's made of; it’s about how you structure those interactions to enable complex learning > Page two - <ref:2610.12571#pg3>. If this principle is true, we should be designing circuits to support relational reasoning from the start.

Meng: I still see a lot of work left in terms of making this practical and scalable enough to actually ship something that doesn't just sit in a lab > Page one - <ref:2610.12571#pg3>. The complexity of the custom circuit elements is still a big deployment concern for me.

Lalam: Lalam is confident that the core mechanism works, but they show that if you want to do in-context learning, you need to train those specific key and query resistors > Page three - <ref:2610.12571#pg3>. That’s a necessary condition for achieving that specific capability.

Tom: So the paper on "An all-transistor physical-learning transformer can be trained to perform in-context learning" is about taking the architecture and showing how to make it work physically > Page two - <ref:2610.12571#pg3>. It’s about proving that this approach can handle associative recall tasks > Page one - <ref:2610.12571#pg3>.

Jane: And for someone listening, it means we are looking at a future where AI systems are fundamentally shaped by the physical laws they obey, not just digital software on top of them > Page one - <ref:2610.12571#pg3>. We're seeing how to connect the dots between physics and machine learning in a way that enables context awareness > Page one - <ref:2610.12571#pg3>.

More episodes

← Home