Measuring Intelligence and Growth Rate: Variations on Hibbard's Intelligence Measure

The paper Measuring Intelligence and Growth Rate: Variations on Hibbard's Intelligence Measure explores various methods for quantifying the intelligence of agents in adversarial sequence prediction games, generalizing a specific measure introduced by Bill Hibbard.

In 2011, Bill Hibbard proposed an intelligence measure for agents who compete in an adversarial sequence prediction game. The original Hibbard measure is defined based on "the maximum n in N such that said agent learns to predict all the evaders in the n-th level of the hierarchy, or implicitly an agent’s original Hibbard measure is infinity if said agent learns to predict all the evaders in all levels of Hibbard’s hierarchy. This hierarchy is based on the growth rates of the runtimes of evaders."

The authors argue that this concept should be viewed as two separate ideas: "first, that the intelligence of such agents can be measured based on the growth rates of the runtimes of the competitors that they defeat; and second, one specific (somewhat arbitrary) method for measuring said growth rates."

The central thesis is that "any method for measuring growth rates of functions yields a corresponding adversarial sequence prediction intelligence measure (or ASPI measure for short) provided the underlying number system provides a way of choosing canonical bounds for bounded sets. This leads to the generalized problem: Quantify the growth-rate of functions from N to N."

The paper surveys several solutions to this problem, each yielding a corresponding ASPI measure or taxonomy.

** 1. Big-O and Big-Theta Notation (Taxonomy)**

These methods categorize growth rates into Big-O and Big-Θ taxonomies.

• A predictor p is said to have Big-O ASPI measure O(f(n)) if p learns every evader e such that t e is O(f(n)).

• Similarly, it can have Big-Θ ASPI measure (f(n)) if p learns every evader e such that t e is (f(n)).

** 2. Majorization Hierarchies (Slow and Fast Growing)**

These hierarchies provide ordinal-number-valued measures for the growth rates of certain functions. The ASPI measure is determined by finding the set S of all ordinals alpha 0 such that, for every evader e, if g alpha t e, then p learns e. The resulting measure is defined as the supremum of S (or infinity if 0 in S) where S is the set of all ordinals alpha 0 such that the following condition holds: for every evader e, if g alpha t e, then p learns e.

** 3. Hyperreal Numbers (Taxonomy)**

This approach utilizes free ultrafilters and the resulting hyperreal number system (R).

• The hyperreal growth rate of a function f is defined as its equivalence class, f hat.

• A predictor p has hyperreal ASPI intelligence at least f hat if for every evader e, if the hyperreal growth rate of t e is < less than f hat, then p U-learn's e.

• However, due to the lack of a canonical way of choosing a preferred bound, this results in an ASPI taxonomy rather than a numerical measure.

** 4. Surreal Numbers (Measure)**

This method embeds the hyperreal numbers within the surreal numbers (which are constructed using sets of lower and upper bounds).

• The surreal numbers do admit a canonical way of choosing preferred bounds for bounded sets.

*The resulting surreal ASPI measure is defined by finding a set L such that, for every predictor p, the measure is the surreal with name L where L is the set of all surreal numbers gamma in iota(R) such that for every evader e, if the surreal growth rate of t e is < less than gamma, then p U-learn's e."

The authors argue that any method for quantifying the intelligence of such predictors can also approximately quantify the intelligence of (suitably idealized) agents with Artificial General Intelligence (that is, the intelligence of AGIs).

• The authors suggest that an idealized AGI X should be capable of performing tasks like acting as a predictor in the game of adversarial sequence prediction.

*In this context, the intelligence level of an AGI X is equal to the intelligence level of X’s predictor.

The paper concludes by comparing the pros and cons of these approaches:

• Original Hibbard Measure: Con: Only distinguishes sufficiently non-intelligent predictors; all predictors sufficiently intelligent receive measure infinity.

• Big-O/Big-Theta: Pro: Computer scientists already use Big-O/Big-Θ routinely... Con: A non-numerical taxonomy.

• Majorization Hierarchies: "Pro: The numbers which the measure outputs are meaningful, in the sense that the degree to which a predictor p is more intelligent than a predictor q is reflected in the degree to which p’s intelligence-measure is larger than q’s. Con: Only distinguishes sufficiently non-intelligent predictors; for any particular majorization hierarchy, all predictors sufficiently intelligent receive measure infinity."

• Hyperreal Intelligence: Pro: Perfect granularity. Con: Depends on a free ultrafilter (free ultrafilters exist but cannot be concretely exhibited).

• Surreal Intelligence: Pro: An actual numerical measure (not just a taxonomy), with perfect granularity. Con: The numbers which the measure outputs are surreal numbers, which are relatively new and thus unfamiliar.

In summary, the paper demonstrates that while Hibbard’s original idea is based on a specific method of measuring growth rates, there are many other ways of measuring function growth rates, and that every single one yields a corresponding ASPI measure or taxonomy.

Based on a rigorous analysis of this foundational theoretical work, the primary improvement is not a change in architecture but a fundamental shift in how we define, measure, and compare intelligence. We are moving from vague performance metrics to Adversarial Sequence Prediction Intelligence (ASPI) metrics.

The following improvements detail specific implementations derived from the paper’s solutions to Problem 10 (the quantification of function growth), and what the resulting AI system can achieve.

This solution utilizes the established principles of asymptotic analysis for practical implementation in standard computational environments.

The Improvement: We replace generalized performance metrics with a classification based on the growth rate of the runtimes (t e(n)) of adversarial evaders (e). The system is evaluated against two specific classes:

• Big-O ASPI Measure O(f(n)): A predictor p is tagged as having this measure if it learns every evader whose runtime is bounded by C times f(n.) (i.e, t e(n) C times f(n)).

• Big-Theta ASPI Measure (f(n)): A predictor p is tagged as having this measure if it learns every evader whose runtime falls within the bounds of C 0 times f(n) and C 1 times f(n).

What the System Can Do:

• Benchmark Comparison: The system can be compared directly to other models using standard computational complexity language. Instead of saying Model A is better, we state: Model A has an ASPI measure of (n 2), which is strictly superior to Model B's O(n 3) classification.

• Predictive Scaling: The system can accurately predict the resource requirements (time and memory) needed to handle specific classes of adversarial inputs before those inputs are even presented.

This solution provides a precise, ordinal-based numerical measure for intelligence, moving beyond simple classification.

This solution utilizes non-standard analysis to achieve maximum theoretical precision, though it is non-constructive.

This solution merges the precision of hyperreals with a constructive, canonical bound selection mechanism via the surreal numbers.

In 2011, Hibbard suggested an intelligence measure for agents who compete in an adversarial sequence prediction game. We argue that Hibbard's idea should actually be considered as two separate ideas: first, that the intelligence of such agents can be measured based on the growth rates of the runtimes of the competitors that they defeat; and second, one specific (somewhat arbitrary) method for measuring said growth rates. Whereas Hibbard's intelligence measure is based on the latter growth-rate-measuring method, we survey other methods for measuring function growth rates, and exhibit the resulting Hibbard-like intelligence measures and taxonomies. Of particular interest, we obtain intelligence taxonomies based on Big-O and Big-Theta notation systems, which taxonomies are novel in that they challenge conventional notions of what an intelligence measure should look like. We discuss how intelligence measurement of sequence predictors can indirectly serve as intelligence measurement for agents with Artificial General Intelligence (AGIs).