Universal Limits on Intelligence
Authorship statement: A human (Reginald Raye) is responsible for 100% of the words and 80% of the argument that follows. The rest can be blamed on Claude Opus 5.
“Superintelligence doesn’t give you a god’s-eye view, it just moves your blind spot.”
Definitions are both overrated and misleading (implying, as they do, a command of the terrain that no map can offer). Nevertheless, you are owed one here. So let us confine our definition of intelligence to an agent1 having the capacity to encode, retrieve, and manipulate information in service of making behavioral decisions2.
This definition, while mainstream, is shot through with holes. I believe the core problem is that any definition that satisfies intuition about the word-concept ‘intelligence’ is inevitably going to be a kludge—that is, it must stitch together disparate faculties along disparate dimensions. Such a result is neither elegant nor satisfying, nor does it have the ring of something universally recognizable, like pi or the speed of light. But, for our purposes here, a tentative working definition is sufficient.
An implication of this definition is that the lower bound on intelligence may be very low indeed. We might describe it as the amount by which an agent’s behavior can undermine its capacity to make the behavioral decisions it would otherwise wish to make—or more simply, self-sabotage. And if we posit that almost all agents exist in rich contexts including other agents, it becomes the collective detriment to behavioral decision-making. On this reading, negative sum games are great vehicles for exploring the lower bound.
The reason I write this note is not because anyone is particularly interested in the lower bound, however. On the contrary, all of cyberspace is abuzz with speculation on intelligence’s upper limit. AI seems to use as fodder for much of its answers here a 2011 Scientific American article on the subject, which is fine, as far as it goes. However, I think the dialogue has passed over a number of crucial considerations3, which has motivated me to throw my hat into the ring.
A final note before we begin: the limits described below are not specific to primate neurology. They follow from universal properties of computation, the relationship between evidence and theory, the ineluctable laws of logic and of physics. They apply to all agents, regardless of whether their intelligence is evolved or engineered. The bottom line, dear reader, is that none of the limits is surpassable by adding intelligence, as they all make up the water intelligence is swimming in.
Computational Limits
If P≠NP, that holds for every possible reasoner. Many of the practical problems facing an intelligence — things like scheduling, optimal planning, and simulation — have solution spaces that grow exponentially with problem size. Every superintelligence worthy of the name has to run searches in such spaces. The heuristics used to do so may be smarter-than-human, but the complexity class of the search spaces is unchangeable. That creates a lot of fallout, even for a superintelligence equipped with the best computer achievable in this universe, like million-year compute durations (durations which often cannot be predicted).
Rice’s theorem dictates that all non-trivial semantic properties of computer programs are undecidable. In general, you cannot determine what an arbitrary program will do without running it. A superintelligence reasoning about other superintelligence (or even itself) is therefore reduced to simulation, the computational cost of which is at least as high as the thing being simulated. An upshot is that mutual transparency between intelligences is not merely difficult. It is, in general, unavailable.
Chaitin incompressibility dictates that almost all bit strings (A.K.A. facts) are algorithmically random, and hence admit of no description shorter than themselves. If you’re interested in describing reality in an economical way — i.e. using compression techniques like formulas and heuristics — you’re going to have a bad time. So many domains of interest — from history to turbulence to ecology — are subject to this computational irreducibility; the best anyone can do is manually enumerate the data fact-by-fact, zero compression. And this assumes, by the way, you even have access to fact-by-fact data, which (unless you’ve been studiously recording the phenomenon of interest) you won’t.
Suppose a superintelligence says “to hell with these headwinds, I will make the universe’s best computer”. They are, heavyweight status notwithstanding, still stymied at every turn. The speed of light puts a ceiling on the speed at which information in the computer can travel. And the bigger they make the computer, the more this transit time bottleneck bites. They also face a size floor — miniaturization cannot proceed past a certain scale (a floor so high that humans have already nearly hit it). Landauer’s principle then sets a severe energy cost on computation, meaning that if they overclock their best-in-the-universe-hardware, it’s still just going to melt. The Margolus–Levitin bound imposes another severe ceiling on operations per unit time. The next victim of physics is the memory system in their computer, which cannot surpass the Bekenstein bound. The list of constraints goes on.
Limits around the size, speed, memory, etc. of information processing are unavoidable. So unavoidable, in fact, that a global optimum cannot exist even in principle. Engineering teams everywhere in the universe have been stuck optimizing over whatever dimensions they care most about, just cursing Klangdür’s principle and Birkenzorg’s bound.
A critical reader might claim that a superintelligence would simply turn to hypercomputation of the sort potentially enabled by Malament-Hogarth spacetime. But, even if hypercomputation can exist, it only partially obviates some of the aforementioned limits. The wall moves back a few paces, yet it remains.
Empirical Limits
No amount of intelligence dissolves Duhem-Quine underdetermination. That is, hypotheses cannot be tested singly. Any prediction derived from a hypothesis is perforce conjoined with auxiliary assumptions about instrument accuracy, background theory, absence of interference, etc. When a prediction fails, logic condemns the conjunction. But the problem is that logic does not suffice to identify the guilty conjunct, meaning that there is inevitably ambiguity about where to absorb an anomaly.
Pushed further, Duhem-Quine underdetermination yields the claim that for any finite body of evidence, there exist mutually incompatible theories that fit it equally well. Quantum mechanics is a standard example: Bohmian, Everettian, and collapse accounts make identical predictions while positing incompatible worlds. A superintelligence can doubtless enumerate such rivals more completely and evaluate them more robustly. Still, the explanation set cannot shrink to a single explanation, and choosing within the set requires criteria (e.g. parsimony, unification, explanatory depth) that evidence does not, cannot, supply.
Another empirical bottleneck not dissolved by intelligence is the problem of induction. That is, every inference from observed cases to unobserved ones assumes that the unobserved resembles the observed. This assumption cannot be proven with deduction, since there is no contradiction in supposing that tomorrow differs. Nor can it be proven with induction, since doing so would require assuming what you set out to establish4.
Worse still, Pearl’s identifiability results are inviolable. Certain causal structures cannot be distinguished from observational data alone, no matter how much of it there is. The only recourse is intervention, which costs time, requires physical access, is bounded by what can actually be manipulated, etc. This applies to even the highest superintelligence achievable in this universe. Seal it in a room with all the data humanity has ever collected, and it still hits a ceiling. The only way to break through it is to leave the room and conduct an intervention.
Data access leads to another inviolable limit. Most of what has happened is gone. Entropy erases the overwhelming majority of the historical record, not through any kind of encryption, but through absolute thermodynamic destruction. Information thus lost is not recoverable. No amount of inferential ingenuity can reconstitute a state whose traces no longer exist. And cosmological horizons place most of the universe permanently out of causal contact.
Chaotic systems are responsible for another flavor of data access constraint. Prediction horizons in chaotic systems extend only logarithmically in measurement precision. A concrete example: each additional day of reliable forecast demands exponentially finer initial data, with quantum indeterminacy setting an absolute floor on how fine that data can be. Once again, we’re faced with a situation where the only recourse for superintelligence is superdata, drawn from super measuring instruments. Since, past a certain point, no such apparatus is physically constructible, long-run prediction of weather, ecosystems, economies (along with most other systems of interest) is bounded independent of compute.
Epistemological Limits
High intelligence requires reasoning about other systems. But any model of a system contained within that system is lossy. The intuitive explanation is that a part cannot fully represent the whole that contains it (no, not even DNA, which codes itself, along with hair color and leg length). Two more formal descriptions of limits on systems bear mentioning. Tarski’s undefinability theorem dictates that no sufficiently strong system can define its own truth predicate. And Löb’s theorem blocks a system from trusting its own proofs in the way it would need to.
The consequences for superintelligence are severe. It can have no guaranteed outside view of its own failure modes, its own priors, or its own values. An incomplete list of other things arbitrarily high intelligence cannot do:
Verify their own soundness
Fully audit why they concluded what they concluded
Rule out that they are systematically wrong in a way that is invisible from where they stand
In sum, superintelligence doesn’t confer a god’s-eye view, it just moves the blind spot.
In response to this, a critic might wonder why disparate superintelligences don’t collaborate, shoring up these blind spots. Still, there is no workaround to be found—even collectives of superintelligent agents are subject to epistemological limits—they are all objects in the world they are reasoning about. This is why a superintelligence cannot self-modify its way out of epistemological dead ends: a system revising itself cannot fully verify that the successor preserves what mattered about the predecessor.
Logical Limits
David Hume observed that arguments in moral philosophy proceed for a while in a descriptive register and then slip, unannounced, into a prescriptive one. This is not so much because of sophistry but because of a simple fact: no set of purely factual premises can entail a normative conclusion. It’s impossible to declare “you ought to do X” without an implicit supporting premise that is itself normative and, therefore, underivable from factual stock. In short, any argument to an ought has an ought already inside it.
This gap does not close under intelligence, not being a gap that stems from reasoning. Reason is instrumental. Given a goal and the facts, it can determine what to do. But it cannot generate the goal. Terminal values — the things wanted for their own sake rather than as means — are inputs to reasoning, not outputs of it.
The upshot is that a superintelligence’s objective function is not necessarily better than a human’s. Nothing about being smart makes what you want more worth wanting. Nick Bostrom’s orthogonality thesis states this formally: intelligence and final goals vary independently, so arbitrary intelligence is compatible with arbitrary goals. The space of possible value systems is vastly larger than the subset humans would recognize as decent. And there is nothing to suggest that a superintelligence’s objective function would overlap with that subset at all.
Maybe the best value system consists in well-structured evaluative processes — an objective function that updates dynamically, resizes its own scope, and observes itself operating — a moral context window that prices in an open-ended range of considerations, weighing each in a manner commensurate with its nature. In short, maybe the best humans already have value systems near the ceiling of human moral performance. Given the light that Hume has cast on this ceiling, that’s not saying much—but it’s not nothing.
It might be assumed, based on all these limits, that a superintelligence is not going to get that far. Well, yes and no. Omniscience is out of the question, as a logical matter. But, frankly, beyond-human-fabricatable computers are not needed to get a lot of mileage. Just better marshaling the information processing resources presently available on Earth could generate absurd amounts of knowledge heretofore unimaginable. And if you spend more time reasoning, you can draw sharper conclusions. Sharper conclusions enable better experiments, which generate better data, about which you can then reason, in a powerful feedback loop. And researchers have not yet scratched the surface of cross-domain analogy, which will unlock vast swathes of the informational landscape. Finally, it would seem but a trivial matter for superintelligence to avoid the pitfalls of human working memory, motivated reasoning, fatigue, and the many other weaknesses with which we’re all too well acquainted.
So, while the wall ringing us in is definitely out there, its contours are cloaked in swirling mist. Whether humans will better discern the wall — indeed, whether we should — remain open questions.
Footnotes
(1) ‘Agent’ should be construed loosely: the greatest share of intelligence tends to be exhibited by agents composed of agents composed of agents, etc. For example, there is an intelligence gradient ranging from the level of a corporation, to the corporation’s employee, to the employee’s mitochondrion.
(2) Perhaps a pleasant implication of this definition is that, the more an agent empowers similarly oriented agents to make the behavioral decisions they wish to make, the more intelligence it evidences. As if cooperation needed any more plaudits than it already gets!
(3) Let’s start with misconceptions. One is that a superintelligence, unencumbered by the drawbacks of human cognition, would rapidly attain omniscience, or something bordering on it. While it must be granted that there is a great deal of room above us, the room does not extend infinitely. Another is that mammalian brains are a poor substrate for the types of computation implicated in high intelligence. This also is false. On the contrary, the evidence has become increasingly clear that human wetware converges on physical limits, whether impelled by the primordial challenge of procuring dinner, or that of modeling other minds (which, in turn, are modeling us modeling them, ad infinitum).
(4) The problem runs deeper than ‘merely’ justifying induction; one cannot, in principle, even specify which regularities count as regularities. All a superintelligence can say is “today is a lot like yesterday, so tomorrow should be a lot like today.”
Reginald’s Recommended Related Readings
Scott Aaronson, “Why Philosophers Should Care About Computational Complexity“ (2011). The best essay on complexity bounds as philosophical constraints rather than engineering details.
Abram Demski and Scott Garrabrant, “Embedded Agency“ (2018). Explores self-opacity, the collapse of the agent/environment boundary, self-reference, Löbian obstacles to self-trust, and the failures of standard decision theory.
David Wolpert, “Physical Limits of Inference“ (2008). The most direct attempt at a unified impossibility result spanning physics and epistemology.
Seth Lloyd, “Ultimate Physical Limits to Computation“ (2000). The most compact treatment of the Bekenstein and Landauer bounds in one place, along with other physics limits cited above.