Von Neumann Networks

In this work, a novel computational model called Von Neumann Networks (VNNs) is introduced, which leverages state-based artificial neurons embedded in cellular arrays to enable these networks to self-engineer their own high-dimensional connectivity and architecture. This framework is significant because it demonstrates that such systems can be computationally universal and capable of learning their own rules and structures through optimization processes like stochastic gradient descent.

The Core Concept: The Von Neumann Neuron

The paper proposes a novel artificial neuron, the Von Neumann neuron, which differs from conventional deep learning neurons by maintaining its own state or role within the network. This neuron possesses a Codd state, denoted as 's', which allows it to toggle its function between two modes: passing on signals (applying an Identity I operation) or making a non-linear decision (applying the activation function σ). The paper states that this state 's' is designated as a learnable parameter. Furthermore, the neuron's state enables network weights to be directly embedded with the array of cells and act as connections or signal neurons etc., instead of only being capable of decision making.

The Network Architecture: Von Neumann Networks (VNNs)

VNNs are constructed by extending neural operators and learning Green’s functions with convolutions on a cellular topology having a diffusion signature. The framework simulates the interaction of neurons with extensions to neural operators and learning neuron signal propagators known as Green’s functions (equivalently point spread functions or PSFs) to cellular topologies with convolutional operators in order to propagate their impulses. The resulting system is described as a discrete system that integrates a partial differential equation of diffusion type analogous to what Johnie predicted.

Mathematical Formulation and Computation

The mathematical formulation involves defining the Chua field C, where each cell c is defined by the Von Neumann neuron. The contribution or response of each cell with input cell ci at some other cell c is given by δpc ´ ciq and depends on the response of the system called its Green’s function G (Arfken and Weber, 2001) as DGpc, ciq = δpc ´ ciq. The forward pass is constructed via the integral: Cpcq = ∫omega Gpc, ciqfpciqdc, which is essentially the forward pass of the network given a suitable function G and equation (1) is applied during back-propagation via automatic differentiation on the computational graph.

Implementation and Architectural Flexibility

The VNN can be implemented using a tensor form of these equations, utilizing convolutions to represent Green’s functions as large kernels, which can then be learned by adapting well-known general convolution operators in frameworks like JAX. The paper presents two forms of VNNs based on whether one assumes an MLP-like structure for inputs and outputs (hyperplanes) or not. This allows the network to learn its own architecture by learning the Codd states 's' directly through stochastic gradient descent, rather than hard-coding them.

Computational Universality and Results

The framework is proven to be computationally universal, with Proposition 1 stating that the cellular machine (CM) is equivalent to a TM that is Turing complete. This universality stems from the fact that the CM features stochastic gradient descent or manual optimization, allowing it to self-engineer its program. Experiments show VNNs can support various tasks, including solving binary arithmetic (XOR operations) and performing 8-bit arithmetic. The ALU model demonstrated the system's ability to use the full d dimensional space of the Chua field C at once, which was predicted by Johnie von Neumann. Furthermore, a tape-based model for XOR arithmetic was shown to be very reminiscent of a Turing machine (TM).

Theoretical Foundations

The theoretical foundation relies on three proofs demonstrating Turing completeness: constructing a universal program/machine that can simulate any program; showing the ability to perform all basic operations leading to such a program; and constructing an actual TM within the system itself. The Cellular Machine (CM) is formally defined as a system where each cell maintains its own state and that their connectivity is defined by a neighborhood rules, and this function, in conjunction with individual cell states, creates the program/transition rules. This structure ensures that the VNN can self-engineer its program.

Limitations and Future Directions

Current limitations include the use of convolution kernel sizes tending to be large (greater than 13). Future work is suggested to explore better support for large kernel sizes, such as scaling methods or dilated approximations. Additionally, exploring quantization of states and adapting the Von Neumann neuron to other topologies where convolution operators exist, such as graphs, is noted as a direction for future research.

Conclusion

The VNN presents a "computationally universal system capable of networks with no discernible layers that can embed a TM with a tape structure and a ‘healable’ Von Neumann computing architecture purely through model training.

Based on the provided scientific paper, here are the specific improvements that can be made to AI systems by implementing Von Neumann Networks (VNNs), along with what these improved systems can achieve:

Specific Improvements for AI Systems

The core improvement lies in shifting from fixed, hand-crafted neural network architectures to a system capable of self-engineering its own topology and connectivity. This is achieved through the proposed Von Neumann Neuron and Von Neumann Network (VNN) framework.

1. Self-Engineering of Architecture (Topology Learning):

2. Learning Dynamic Connectivity: The VNN framework allows the network's architecture—specifically, which neurons connect to which—to be learned directly through stochastic gradient descent by optimizing the learnable Codd states and connection weights, rather than relying on pre-defined structures like trusses or fabrics.

3. Integration of Computational Universality: The system is mathematically proven to be computationally universal (Turing complete), meaning it can simulate any computation that a standard computer can perform.

What the Improved AI System Can Do

The resulting VNN-based AI systems possess capabilities far beyond current fixed architectures, enabling them to perform complex, adaptive computational tasks:

1. Learning Novel Architectures for Specific Tasks:

2. Task-Specific Optimization: The network can automatically determine the optimal structure (hyperplane vs. full connectivity) and connectivity required to solve a specific objective function or task without requiring manual architectural design.

3. Universal Computational Modeling (ALU Simulation):

4. Arithmetic Logic Unit (ALU) Functionality: The VNN can learn to simulate an ALU directly, demonstrating its ability to handle complex arithmetic and logic operations within its cellular structure, potentially surpassing the performance of traditional MLPs on specific computational problems (as shown in the results).

5. Self-Replication and Program Generation:

6. Turing Machine Emulation: The system can be configured to emulate a Turing Machine (TM) architecture using its learned Green's functions as transition rules, effectively allowing the AI to learn and execute arbitrary programs on a tape structure embedded within the cellular array.

7. Adaptive Pattern Recognition:

8. Arbitrary Input/Output Mapping: The VNN can support arbitrary input/output shapes (not just standard vector inputs) by dynamically adjusting its connectivity, making it suitable for complex data structures beyond simple image or tabular data formats.