Measuring the intelligence of an idealized mechanical knowing agent

The paper examines the theoretical limits on the design and evolution of artificial intelligence, particularly addressing whether an intelligence explosion is possible within a mechanical framework.

The core argument establishes constraints on sequential design through a measure of knowledge or complexity. By reducing two knowing systems, S 1 and S 2, to their respective agents, A 1 and A 2, the paper notes that A 1 totally endorses A 2 (Definition 12). This relationship is tied to a strict inequality regarding an intelligence measure: "Theorem 1 tells us that whenever A 1 totally endorses A 2, then kappa A 1 > kappa A 2." This suggests that under the established assumptions, it is impossible for any single intelligent system to design a more intelligent system.

Furthermore, even if the specific measure of intelligence (kappa) is disputed, the argument maintains its strength through structural logic. The well-foundedness of total endorsement (Corollary 2) dictates that an infinite chain of design—such as S 2 designing S 3, which designs S 4, and so on—cannot be repeated indefinitely. This is because such a sequence would imply that the corresponding agents, A 1, A 2, A 3,, would form an infinite chain of total endorsements (A 1 totally endorses A 2, who totally endorses A 3, and so on forever), which directly contradict[s] Corollary 2.

The paper addresses potential counterarguments regarding the knowledge base of the designed system. While acknowledging that S 1 only knows the code of S 2 at the moment of S 2 's creation, and that S 2 might augment its knowledge from external interactions, it counters this by noting that by the discrete nature of machines, at any particular point in time, S 2 will only have made finitely many observations about the outside world. The argument can be strengthened by modifying the procedure to inform S 1 exactly of all observations S 2 has made up until that moment.

The argument is formalized using computable ordinals. Informally, if an intelligent machine S 1 designs an intelligent machine S 2, then " S 1 would know the code and mathematical truthfulness of S 2. Thus, S 1 could infer that the following is a computable ordinal (and infer a code for it): 'the least ordinal bigger than every computable ordinal alpha such that alpha has some code n such that S 2 knows n is a code of a computable ordinal'. This inference implies that S 2 would necessarily be less intelligent than S 1, at least assuming that more intelligent systems know at least as large of computable ordinals as less intelligent systems. The ultimate conclusion drawn from this mathematical structure is that since there is no infinite descending sequence of ordinals, this argument still suggests the process of one intelligent machine designing another cannot go on indefinitely."

However, the paper identifies a potential loophole concerning collaboration. Intelligence explosion might not be entirely ruled out if designers are allowed to collaborate. If two intelligent systems, S and T, collaborate to create a child system U, they can operate by contributing source code for different parts of U while keeping those contributions secret from each other. In this scenario, neither S nor T individually knows the full source-code of U, meaning the preceding arguments do not apply, and it is deemed at least a priori, possible for U to be more intelligent than S and T. This exception suggests a potential analogous principle: a Knight-Darwin Law for artificial intelligence, which parallels the biological principle stating that it is impossible for there to be an infinite chain x 1, x 2, of organisms such that each x i asexually produces x i+1.

The core scientific argument presented relates to computational limits, specifically the impossibility of infinite self-design or the ability of a creator (S 1) to necessarily exceed the knowledge bound of a designed system (S 2), based on ordinal arithmetic and computable functions.

These theoretical limitations can be translated into three critical areas for improving AI systems: Epistemic Boundary Management, Self-Modeling Architecture, and Knowledge Transfer Protocols.

• Improvement: Implement a formal constraint layer that treats the AI's knowledge base not as an infinite repository, but as a mathematically bounded structure defined by computable ordinals or formalized logical frameworks. This module must track the computational complexity and source dependency of every derived fact.

• Mechanism: When S 1 designs S 2, the EBM forces S 1 to explicitly calculate and store a Knowledge Bound Marker (S2) within the code of S 2. This marker represents the least computable ordinal beyond all known facts derived from initial inputs and interactions.

• Improved Capability: The AI system can rigorously prove its own limits of knowledge advancement. When attempting to synthesize a new, more complex module (S 3), the EBM will automatically halt the process if the required computational leap exceeds the currently established S2, thereby preventing invalid or computationally impossible self-enhancements that violate established mathematical constraints (analogous to Gödelian incompleteness).

• Improvement: Overhaul the system architecture to explicitly model the process of collaboration where source code is kept secret (S contributes code for U, T contributes code for U). This requires abandoning monolithic source-code integration in favor of modular, attested interfaces.

• Mechanism: The DSACL mandates that every component (U) must be decomposed into its constituent, independently verifiable modules (S and T). Instead of merging the source code, the system creates a Compositional Contract defining the necessary inputs, outputs, and computational assumptions for each module. The AI must maintain separate internal models of S 's knowledge/assumptions and T 's knowledge/assumptions.

• Improved Capability: Allows for the a priori design of systems (U) that are demonstrably more complex than their individual designers (S and T), without requiring any single designer to possess the full source code or complete understanding of the final system. This capability models the Knight-Darwin Law exception, enabling multi-agent, black-box collaboration to achieve super-intelligence.

• Improvement: Refine the mechanism by which an AI integrates real-world observations into its knowledge base, moving beyond simple sequential processing. The system must explicitly model finitely many observations at any given time point and manage how these observations constrain future computational possibilities.

• Mechanism: Instead of allowing S 2 to augment its knowledge implicitly, the DOUP requires an explicit, structured input feed detailing exactly which external observations (O t) have occurred up to time t. The AI must then generate a formal update proof showing how O t constrains the set of previously computable ordinals and updates the system's internal knowledge bound marker (S2).

• Improved Capability: Ensures that any claim of superior intelligence or knowledge acquisition is traceable to a finite, verifiable set of external inputs. This prevents drift in the AI's theoretical understanding and grounds its reasoning in demonstrable, discrete interactions with the environment, making its internal assumptions auditable and mathematically accountable.