#17: Seriously, what is Intelligence?
We are in the earliest innings of an intelligence revolution: progress in the field of Artificial Intelligence is now rapid, and the innovations are becoming accessible to the public just as quickly. Millions of people are now using tools like ChatGPT, and may be thinking about what it would mean to have AGI in their lifetimes. This includes fearful policymakers and lobbyists who want to control and restrict AI. They make arguments about the dangers of rogue intelligence — AI escaping our control.
But these arguments tend to leave out what exactly they mean when they say “intelligence”. This is no coincidence, because these arguments are often not actually about intelligence when you examine them closely. They are about related, but distinct concepts (like consciousness or emotion) that are often conflated for anthropological reasons. In that way, because intelligence is such a nebulous concept, it is easy for people to make misleading arguments about it. My objective in this blog post is to:
Explain why intelligence is often conflated with other concepts;
Try to disentangle intelligence from these concepts;
Discuss what the current generations of LLMs are showing us about intelligence;
Predict what the next generations of LLMs may look like, and what we may learn about human theory of mind in the process.
Anthropic Bias
Out of all the life on planet Earth, we humans are spectacularly unique and capable. For example, we possess the following qualities:1
Intelligence: the ability to perceive, infer, acquire, retain, and apply information
Consciousness: the awareness of our own existence
Emotion: the capacity to alter mental state due to physiological change;
Reason: the ability to logically draw true statements from other true statements;
This is in contrast to all other animals, which either do not possess these qualities at all, or to vastly smaller extents.2 This leads to what some call Anthropic Bias:3 for example, if we view ourselves as the only truly intelligent creatures, and as the only truly conscious creatures, then it is tempting to think that intelligence and consciousness are tied together; perhaps even that they are the same thing.
This is the conflation that I mentioned earlier: people talk about intelligence when they really mean reason or consciousness. From a normal human perspective, because these qualities are so intertwined to us, they get lumped together as one. But they shouldn’t be, and contemporary progress in LLMs makes this clear.
What We’re Learning from LLMs
If you’ve played with ChatGPT, your immediate response was probably something like: wow, this is intelligent. There’s no doubt it passes a 1980s-style Turing Test. You can ask it about the significance of the Peace of Westphalia or what a Monad is, and the answers will be better than what I can tell you.
But what’s also apparent from playing around with ChatGPT is that it doesn’t seem to meet any standard of emotion or consciousness. In fact, I’d argue it’s not even capable of reasoning: it fails to compute simple arithmetic. Furthermore, the core architecture of ChatGPT — it’s a transformer-based LLM — is already at huge scale: it’s trained on a dataset that most likely constitutes a significant percentage of all human text in existence, so it’s not clear whether scaling it up another 10x or 100x will cause its abilities to converge toward “true” reasoning. Reading the tea leaves of the contemporary literature reinforces my impression: precise reasoning may require some different or additional architecture.4
In the previous section, I listed “intelligence” and “reason” as separate bullet points, which may have looked unusual. This is the reason I did so: one of the things that’s been enormously surprising to me in my work with LLMs is that conventional information-learning-and-recall notions of “intelligence” may be mostly distinct from the ability to reason.5 Fascinating!
So What is an LLM?
Well, in the context of artificial intelligence, it doesn’t really resemble humans at all. It’s a totally new thing. No consciousness, no emotion, arguably no reasoning, just pure informational intelligence in the form of a gigantic interpolative datastore.
In theory of mind, there is the popular thought experiment of a P-Zombie, short for a “philosophical zombie”: a creature that is externally identical to a human, but does not have an internally conscious experience. For example, if you were to poke a P-Zombie with a stick, it would say “ouch” and “get that away from me” and step backwards, but it would not internally feel pain or think “I will tell my sibling about the maniac with the stick” as a person might.
Similarly, if you were to prompt ChatGPT with the phrase “your beloved family pet has died. How do you feel?” and then “now tell your teenage daughter about this,” you will get perfect human outputs: a somber reflection on mortality, an expression of grief, and a delicately worded parental message. It reads like emotion, but it isn’t. ChatGPT does not have a dog or a family or a daughter or continuously running internal dialogue,6 all of this is just P-Zombie simulacrum.
Dismantling AI Safety Arguments
Now that we’ve built up some precision in discussing intelligence, disentangled it from other notions like consciousness, and pointed out some of the singular weirdness and limitations in LLM behavior, we’re better-equipped to cut through the noise of some AI Safety arguments. Specifically, the broad class of argument that I’d like to address goes something like this:
Encounters between species or civilizations with a great imbalance of power tend to end in the destruction or conquest of the weaker one. If we invent a digital super-intelligence that is hostile to us, we would be outmatched and face existential risk.
There are a lot of subtly-buried assumptions in there that make this argument misleading:
The analogy to inter-species or inter-civilizational conflict is clearly inappropriate. While it might be temping to look at AI through the interpretative lens of things that we know well — ourselves — it doesn’t fit. We have shown that LLMs are nothing like any existing thing.
Hostility or conflict of any kind follows some impulse for self-preservation, and acting on that, subject to an environment of scarce resources and zero-sum competition. This is natural to virtually all animals on planet Earth (including ourselves): Darwinian evolution results in creatures that protect themselves and reproduce. This is such a powerful historical filter that this trait is deeply ingrained in us, but a priori there is no reason why any kind of artificial intelligence — not subject to the same evolutionary filter at all — would share this characteristic.
What does “super-intelligence” mean? Would that be a thing that is extremely good at learning, retrieving, providing, and applying information? Doesn’t GPT-4 already meet that definition? The conflation is that “intelligence” here implies the capability for reasoning too — but we still seem to be very far away from that.
Being Precise about Intelligence
This is not the first time in history that a widely-used concept, when closely examined, has turned out to be opaque and instead give way to an assembly of many other concepts: the ancient Greeks called everything philosophy; this has now been broken down into hundreds of scientific disciplines. In the 20th century, psychologists diagnosed everything as schizophrenia; today that’s an exceedingly rare diagnosis, but the DSM-5 lists over 300 more precise conditions.
The same may occur for intelligence: as the field of artificial intelligence keeps advancing and we work practically with more forms of AI, we will be pressed to split many of these nebulous, broad concepts up into precise characteristics. This may cause us to re-shape our own theory of mind as well. It seems likely to me that in a few years, the model of human cognition will not be as a monolith, but rather as the outcome of many distinct systems interacting in complex and perhaps surprising ways.78
Continued research in LLMs may take us to an unprecedented position, where we will have invented these things that are hyper-intelligent in a very specific way — extraordinarily powerful at statistically compressing knowledge and returning approximations to it — but otherwise totally inert. This breaks how most people currently think about intelligence, and that’s why I wrote this blog post — it’s important to start thinking differently about these constructs if you don’t want to make premature or dumb policy decisions.
The Intelligence Revolution
I wrote at the start that we are in the very earliest stages of an intelligence revolution. What did I mean by this?
Consider the industrial revolution. Some historians define it by the inventions of certain machines and processes — but others see it as defined by a break from history: the amount of energy harnessed and consumed by humanity began to increase super-linearly per capita. The availability and energy density of fossil fuels was a tremendous unlock, probably a necessary condition for bringing our civilization to where it is today.
Similarly, I have little doubt that future historians looking back on our time will again see it as defined by a break from history: the amount of intelligence available to humanity began to increase super-linearly per capita in the early 2020s, and will have been a tremendous unlock. But it’s yet unclear what exactly that intelligence really comprises, or what the retrospective measurement of it will be.9
This is not an exhaustive list of qualities that make us distinct or are relevant to consider when discussing intelligence. For example, notions like the capacity for the subjective-objective distinction or self-motivation are worth taking into account; they are just not as core as the main four that I mentioned above.
For example, great apes are commonly understood to exhibit some intelligent traits — just much less so than humans.
I know that there is a book related to this topic called Anthropic Bias by Nick Bostrom. I have not read it. I am using the term Anthropic Bias strictly because it is useful and precise, not because I am trying to provide any kind of literature reference.
The connection between Vishal Misra’s paper and symbolic reasoning may not be immediately apparent. Here’s how I think about it: Misra’s paper shows that LLMs are not capable of recursive self-improvement. But symbolic reasoning abilities would enable self-improvement: logical reasoning enables the creation of new true statements based on existing true statements, and the refinement of internal knowledge — i.e. using reasoning to evaluate truthfulness of internal statements and the discarding of ones that are proven false. But if LLMs cannot recursively self-improve, then by contraposition it must be true that LLMs are not capable of logical reasoning either, which would otherwise enable recursive self-improvement.
To the core point of this article, note of course that this observation is relying on a particular definition of intelligence, which is up for debate.
This is a total aside: I’ve been thinking about implementing LLMs with continuously running internal monologue and seeing how that changes their behavior. One of the obvious big differences between LLMs and brains is that brains are always on and subject to some quasi-random electrical perturbations, whereas LLMs lie dormant in-between function calls. If you’d be interested in putting together some after-work experiments in LLM behavior with me, please email me at contact@johnloeber.com.
Scientific findings about the impact of our microbiotic flora and gut-brain axis over the last decade seem like an early indicator of the type of systemic discoveries yet to come.
Perhaps implicitly, this observation is why there has been a big resurgence of interest in the work of Julian Jaynes over the past year. I recommend reading Scott Alexander’s commentary on it.
Unlike energy, which is easily measured in Joules, it is not clear today what the best measure of this computational intelligence is. I think this is funny, because people in my field love to say they’re building “toward intelligence too cheap to meter” — what unit would you meter, anyway? Gigahertz? Floating-point operations? If it is actually too cheap to meter, the question is of course moot, but it won’t be for the foreseeable future.