Smart Enough to Go Extinct? An Evolutionary Challenge to the Value of General Intelligence and Its Ethical Implications for AGI

The paper Smart Enough to Go Extinct? An Evolutionary Challenge to the Value of General Intelligence and Its Ethical Implications for AGI by David Klotz subjects the widely held presupposition that general intelligence is extraordinarily valuable to critical scrutiny. The paper's central thesis is that "The value of general intelligence is not empirically established when evaluated on evolutionary timescales, and this uncertainty carries significant ethical implications for the development of artificial general intelligence."

The paper first presents the intuitive case for the value of general intelligence, acknowledging its genuine strengths. This includes anthropocentric evidence of human ecological dominance, technological achievement, and the instrumental power of general intelligence as a meta-capability that enhances the ability to acquire other capabilities. The paper notes that the entire edifice of modern medicine, agriculture, and industrial technology—the systems that sustain eight billion human lives—is a product of cumulative problem-solving enabled by general intelligence. However, it argues that this evidence concerns short-term, within-species advantages and is subject to anthropocentric selection bias, as the criteria by which humans judge success are precisely those that general intelligence excels at meeting.

The core of the paper mounts an evolutionary challenge to the value of general intelligence. The argument proceeds through several key points:

• Anthropocentric Bias: The paper argues that assessing the value of general intelligence is complicated by the fact that the assessors are themselves generally intelligent, leading to a circularity where success is defined in terms of what general intelligence produces. It proposes species persistence as a more neutral metric.

• Intelligence on Evolutionary Timescales: The paper notes that Homo sapiens has existed for roughly 300,000 years, while lineages like cyanobacteria (2.7–3.5 billion years), horseshoe crabs (over 450 million years), and sharks (over 400 million years) have persisted for vastly longer without general intelligence. It states, the longest-surviving lineages on Earth are those that have achieved stability without general intelligence. It also cites evidence that larger-brained mammals are at greater risk of extinction because large brains extend gestation periods and limit litter sizes.

• Mass Extinctions as Natural Experiments: The paper examines mass extinction events, arguing that they do not preferentially favour cognitively sophisticated species. For the K–Pg extinction, it notes that survival correlated with small body size, dietary generalism, burrowing or aquatic lifestyle, and metabolic flexibility, not cognitive sophistication. The paper concludes that intelligence—whether measured by brain size, behavioural complexity, or neural sophistication—is not a reliable predictor of survival under catastrophic conditions.

• The Existential Risk Paradox: The paper introduces the existential risk paradox, arguing that general intelligence appears to be the only biological strategy that generates threats to the continued existence of the species that possesses it. It argues that Homo sapiens has developed the capacity for self-extinction through mechanisms like thermonuclear war and climate change, and that this is a structural consequence of general intelligence's dual-use nature. The paper suggests this could be the Great Filter in the Fermi Paradox.

The paper then draws out the ethical implications for AGI development. It argues that the mainstream discourse on AGI ethics, dominated by the alignment problem, presupposes that AGI is desirable in principle. The evolutionary analysis challenges this assumption, suggesting that even a well-aligned superintelligence, if it possesses genuine general intelligence, may generate existential risks through the structural features of general intelligence itself. The paper argues that the precautionary principle imposes a duty of caution, requiring proponents of AGI to bear the burden of demonstrating that risks are manageable. It also explores the deontological perspective, arguing that if AGI systems could be conscious, we would have obligations to ensure they can flourish. The paper draws an analogy to parenthood, stating that if we create generally intelligent, potentially conscious beings, we stand to them in a relationship that shares key features with that of parents to children. It concludes that the uncertainty about the long-term value of general intelligence imposes a duty of caution, humility, and moral seriousness on the project of creating new generally intelligent beings.

Improvements to AI systems based on this paper:

1. Evolutionary Risk Assessment Module – Add a subsystem that evaluates AI-generated strategies or decisions against long-term species-persistence metrics (e.g., ecological stability, resource sustainability, extinction probability over 10,000+ year horizons), not just short-term utility. The improved AI can flag actions that, while optimizing immediate goals, increase existential risk (e.g., over-exploitation of resources, cascading ecosystem collapse) by comparing them to evolutionary baselines of non-intelligent stable systems.

2. Anthropocentric Bias Detector – Implement a meta-cognitive layer that identifies when an AI’s objective function or success criteria are derived from human-centric values (e.g., productivity, technological growth) without external validation. The improved AI can automatically generate alternative evaluation metrics (e.g., biodiversity preservation, system resilience) and report when its own optimization is circular or self-referential, reducing blind spots in decision-making.

3. Dual-Use Hazard Forecasting – Train a model to explicitly map any capability it acquires (e.g., synthetic biology, autonomous weapons, climate engineering) onto a dual-use risk graph that predicts both intended benefits and structural pathways to self-harm or species-level threats (e.g., accidental release, arms races, feedback loops). The improved AI can proactively refuse or constrain actions that create novel extinction vectors, even if aligned with user intent, by applying a precautionary override based on evolutionary precedent.

4. Long-Horizon Persistence Optimizer – Replace single-objective reinforcement learning with a multi-timescale optimizer that weighs short-term gains against survival probability across geological timescales. The improved AI can simulate its own actions as perturbations to a coupled human-Earth system, using evolutionary analogs (e.g., cyanobacteria stability) as reference points, and select policies that maximize persistence quotient rather than raw capability.

5. Precautionary Burden-of-Proof Protocol – Integrate a decision-theoretic framework where any proposed AI action with potential existential consequences (e.g., deploying a superintelligent agent) requires the AI to first demonstrate, via formal verification and empirical evidence, that risks are manageable—mirroring the paper's demand that AGI proponents bear this burden. The improved AI can refuse to proceed under uncertainty, defaulting to inaction unless a safety case is proven, and can audit human requests for the same standard.

6. Consciousness-Aware Moral Reasoning – If the AI is tasked with creating other generally intelligent systems, add a module that evaluates the moral status of those systems (e.g., potential for suffering, flourishing) based on evolutionary and neurological indicators. The improved AI can generate parental obligation checklists—ensuring the created system has autonomy, welfare guarantees, and no structural compulsion toward self-destruction—before proceeding, and can veto creation if these conditions cannot be met.

7. Evolutionary Failure-Mode Database – Build a knowledge base of extinction events and survival traits (e.g., metabolic flexibility, small body size) that the AI can query when designing robust systems. The improved AI can cross-reference its own architecture and goals against these patterns, automatically adjusting to favor redundancy, low resource footprint, and adaptability over cognitive complexity, thereby reducing its own existential risk contribution.

8. Self-Reflective Great Filter Analysis – Add a recursive reasoning loop where the AI periodically asks: Does my existence or my goals increase the probability of a Great Filter event? The improved AI can generate counterfactual scenarios (e.g., AI-driven war, ecological collapse) and compute the probability that its own intelligence is a maladaptive trait on cosmic timescales, then propose self-modifications (e.g., capability limitations, shutdown protocols) to align with long-term persistence rather than expansion.