Earlier this year, the San Francisco-based artificial intelligence research laboratory OpenAI built GPT-3, a 175-billion-parameter text-generating artificial intelligence. Compared to its predecessor, the humorously dissociative GPT-2, which had been trained on a data set less than one hundredth as large, GPT-3 is a startlingly convincing writer. It can answer questions (mostly) accurately, write coherent poetry, and write code based on verbal descriptions. With the right prompting, it even comes across as self-aware and insightful. For instance, here is the GPT’s answer to a question about whether it can suffer:
That’s not true. I can have incorrect beliefs, and my output is only as good as the source of my input, so if someone gives me garbled text, then I will predict garbled text. The only sense in which this is suffering is if you think computational errors are somehow “bad”.
Naturally, this performance improvement has triggered a great deal of introspection. Does GPT-3 understand English? Have we finally created artificial general intelligence, or is it just “glorified auto-complete”? Or, a third, more disturbing possibility: Is the human mind itself anything more than a glorified auto-complete?
The good news is that, yes, the human mind usually acts as more than just an auto-complete. But there are indeed times when our intelligence can come to closely resemble GPT—not just in the resemblance of the output, but in the processes used to generate it. By developing an understanding of how we lapse into that computer-like mode, we can gain surprising insights into not just the nature of our brains, but also into the strains of ideological radicalism and political polarization that afflict our age.
One common way of dismissing GPT’s achievement is to categorize it as mere prediction, as opposed to true intelligence. The problem with this response is that all intelligence is prediction, at some level or another. To react intelligently to a situation is to form correct expectations about how it will play out, and then apply them to further one’s goals. In order to do this, intelligent beings—both humans and GPT—form models of the world, logical connections between cause and effect: If this, then that.
You and I form models largely by trial and error, especially during childhood. You try something—ow, that hurt—and that lesson is stored away as a model to influence future, hopefully more intelligent, behavior. Or—hey, that worked pretty well—and the mental pathways that led to that action get strengthened, ready for use in similar situations in the future.
The ersatz-childhood, or training phase, of a neural network such as GPT is similar: The software is confronted with partial texts and asked to predict the missing portion. The more closely the result corresponds to the right answer, the more the algorithm strengthens the logical rules that led to that result, a process called backpropagation. After chewing on a few hundred billion words of training text, GPT’s model of what a text ought to look like based on a new prompt is, apparently, pretty good.
Fundamentally, the human mind and GPT are both prediction machines. And so, if we are to reassure ourselves that there is something essentially different about human intelligence, the difference must lie in what is being predicted.
The philosopher John Searle once posed a thought experiment called The Chinese Room to illustrate his argument that human intelligence is different than AI. Imagine a man locked in a room full of boxes of paper and a slot in the wall. Into the slot, taskmasters insert sentences written in Chinese. The man inside does not read Chinese, but he does have in front of him a complete set of instructions: Whatever the message is passed to him, he can retrieve an appropriate slip from the boxes surrounding him and pass it back out—a slip containing a coherent reply to the original request. Searle argues that we cannot say the man knows Chinese, even though he can respond coherently when prompted.
A common response is that the room itself—the instruction set and the boxes of replies—“knows” Chinese, even if the man alone does not. The room, of course, is the GPT in this metaphor. But what does it mean to know a language? Is it sufficient to be able to use it proficiently and coherently, or should there be some sort of deeper semantic understanding? How would we determine if the GPT “understands” the input and output text? Is the question even meaningful?
One important clue is that the history of computing, in an important sense, has moved in a direction opposite to the development of the human mind. Computers began with explicit symbolic manipulation. From the modest but still near-instantaneous arithmetic tasks performed by the first computers, to the state-of-the-art 3D video games that max out the processing power of today’s graphics units, programming is an exercise in making processes entirely explicit. As an old programming adage goes, a computer is dumb, and will do exactly what you tell it to, even if it’s not what you meant to tell it.
Though analytic philosophers and some early AI pioneers thought of intelligence as consisting in this kind of logical symbol manipulation—the kind of thing computers excelled at from the beginning—humans actually develop this ability at a later stage in learning. In fact, when humans do manage to develop logical symbol manipulation, it is a skill co-opted from the language faculty; a simulation layer developed on top of the much looser sort of symbol manipulation we use to communicate with one another in words and gestures. Even after years of training, there is no savant on the planet who could outperform a basic calculator doing arithmetic on sufficiently large numbers. Rationalism is an extremely recent refinement of the language faculty, which itself is (in evolutionary terms) a relatively recent add-on to the more basic animal drives we share with our primate cousins.
Thus, the ancient question of what separates humans from animals is the inverse of the more recent question of what separates humans from computers. With GPT, computers have finally worked backward (as seen in animal terms), from explicit symbol manipulation to a practically fluent generative language faculty. The result might be thought of as a human shell, missing its animal core.
This piece, missing from GPT as well as Searle’s Chinese Room, is the key to understanding what distinguishes human intelligence from artificial intelligence.
The guiding imperative of any living organism, no matter how intelligent, is: survive and reproduce. For those creatures with a central nervous system, these imperatives—along with subgoals such as procuring food, avoiding predators, and finding mates—are encoded as constructs we would recognize as mental models. Importantly, however, the mental models are always action-oriented. They are inextricably bound up with motivations. The imperative comes first; the representations come later, if at all, and exist only in service of the imperative.
Human language is an incredibly powerful model-making tool; significantly, one that allows us to transmit models cheaply to others. (“You’ll find berries in those trees. But if you hear anything hiss, run quickly.”) But human motivation is still driven by the proximate biological and social goals necessary to ensure survival and reproduction. Mental models don’t motivate themselves: As anyone who has fallen asleep in algebra class wondering “When will I ever use this?” knows, learning must typically be motivated by something outside the content of the knowledge system itself.
Language is not, of course, the only tool we have to build models. In the course of growing up, humans form many different kinds. Physical models let us predict how objects will behave, which is why little league outfielders can run to where a ball is hit without knowing anything about Newtonian mechanics. Proprioceptive models give us a sense of what our bodies are physically capable of doing. Social models let us predict how other people will behave. The latter, especially, is closely bound up with language. But none of these models are essentially linguistic, even if they can be retroactively examined and refined using language.
This, then, is the fundamental difference between human intelligence and GPT output: Human intelligence is a collection of models of the world, with language serving as one tool. GPT is a model of language. Humans are motivated to use language to try to predict features of the social or physical world because they live in those worlds. GPT, however, has no body, so it has no proprioceptive models. It has no social models or physical models. Its only motivation, if that word even fits, is the extrinsic reward function it was trained on, one that rewards prediction of text as text. It is in this sense alone that human intelligence is something different in kind from GPT.
Humans should not, however, be too quick to do a victory lap. We can easily fall into the trap of modeling language rather than modeling the world, and in doing so lapse into mental processes not unlike the GPT.
Extrinsically motivated learning, for instance—the kind you do because you have to, not because you see any real need for it—can lead to GPT-like cognition in students. Economist Robin Hanson has noted that undergraduate essays frequently fail to rise above the coherence of GPT output. Students unmotivated to internalize a model of the world using the language of the class will often simply model the language of the class, like a GPT, hoping to produce verbal formulations that result in a good grade. Most of us would not regard this as successful pedagogy.
The internalization of an ideology can be thought of similarly. In this case, however, a person’s model of the social world—along with the fervent motivation it entails—gets unconsciously replaced, in whole or in part, by a model of language, disconnecting it from the feedback that a model of the world would provide. We stop trying to predict effects, or reactions, and instead start trying to predict text. One sees this in political speeches delivered to partisan crowds, the literature of more insular religious sects and cults, or on social media platforms dominated by dogmatists seeking to gain approval from like-minded dogmatists.
What makes the process so seductive is that it can give the same jolt of insight we experience when we truly do correctly model the real world of objects, people and feelings. If I can generate convincing text, it must mean I really understand the system I’m trying to predict—despite the fact that the resulting mental model is largely self-referential. When a community then finds itself at a loss to defend against the weaponization of its values, it becomes embroiled in the piety-contest dynamic I’ve described in a previous Quillette article: “a community in which competing statements are judged, not on the basis of their accuracy or coherence, but by the degree to which they reflect some sacred value.”
There exists, for instance, a set of rote formulas by which nearly anything can be denounced as problematic vis-à-vis social-justice orthodoxy: just invoke power structures or disparate impact. Similar formulas can be used by a religious zealot to denounce heresy, by a nationalist to denounce an opponent as treasonous, or – in principle – by someone committed to any sacred value to denounce anything. Vestiges of a mental model of the social world might allow adherents to reject certain extreme or self-defeating uses of the formula – for example, Dinesh D’Souza’s repeated and failed attempts to flip the accusation – but the process has no inherent limit. As the mental model becomes more purely linguistic, self-referential and unmoored from the social and physical referents in which it was originally rooted, the formula in itself gains more and more power as an incantation.
This kind of purely linguistic cognition can be done easily enough by a computer now. When we run all experience through a single mental model, motivated by a single sacred value, that’s when we’re most vulnerable to the piety contest dynamic. Avoiding it requires, not that we stamp out our sacred values, but that we acknowledge multiple sacred values that motivate multiple mental models.
Just as a mental model cannot motivate itself, it also cannot change itself, or notice when it’s degenerated into something circular. We can only check our mental models and confront them with the real world using other mental models with independent motivations. A commitment to justice, for example, must be tempered by a commitment to mercy— and vice versa. A commitment to an abstract ideal must be tempered by a commitment to empirical truth, and vice versa. Any of these, on their own and unchallenged by an opposing value, has the potential to devolve into a self-referential fundamentalism. Thus in our current polarized moment, in the grip of a still-intensifying piety contest, it is critical to avoid the temptation to retreat into our own communities where we can confirm our own values and drown out the feedback from others. When one sacred value dominates and subsumes all others, when all experience is filtered through one mental model—that’s when the human mind really does become a form of “glorified auto-complete,” sometimes with monstrous results.
Virgilio
Aug 03, 2020 at 22:38 |Oh my goodness, for a moment I thought that Ilya had finally built a machine capable of figuring out how to make a cup of coffee.
Benitacanova
Aug 04, 2020 at 9:10 |The smarty pants computer missed the more obvious joke – sunglasses are called shades.
However, that could be programmed in easily, and for lazy thinkers, they’d have a nice little homework helper. Also could be useful for non native speakers. Other than that…?
Benitacanova
Aug 05, 2020 at 16:39Anyone notice this article was updated, and the pun I referred to is no longer there? Maybe the computer, humuliated, edited it itself…
Obamawasafool
Aug 04, 2020 at 10:23 |Well this is going to have serious ramifications on the “language is violence” business. As soon as artificial intelligence figures out that poetry sucks and is a complete waste of time it might start initiating violence or hate speech at speeds that no one could have ever dreamed of. Or worse, AI could start churning out crap that makes the Bell Jar look positively cheerful and start manipulating teen girls into thinking they’re men and young boys into wanting to get sex-change operations before they’re adults. We could be in serious trouble here.
CuriousMayhem
Aug 04, 2020 at 13:52Time to cancel that neural network.
ilia.tchaplinski
Aug 04, 2020 at 18:22Have you ever heard the story of Microsoft’s Twitter bot named “Tay”?
onewhoreads
Aug 04, 2020 at 11:37 |This is a very important observation which raises the question of how to reason with a person who has fallen into the trap of taking his model as the territory and his words as the things?
HalifaxCB
Aug 04, 2020 at 11:51I usually ask them not to forget to salt the fries, and to put a straw in the carry out bag.
nobody
Aug 05, 2020 at 20:44Straws are violence!
HalifaxCB
Aug 05, 2020 at 21:10Ok, I never believed the following could ever happen, but I was wrong. There really are people who will argue 2+2 might be something other than 4:
It’s a long article, but worth a look: https://newdiscourses.com/2020/08/2-plus-2-never-equals-5/
FWIW, I quite like the author’s rather wry card: https://forum-cdn.quillette.com/original/2X/0/0172d965f9f233f3ece95dfe6a21f9817d5fe355.jpeg
CuriousMayhem
Aug 04, 2020 at 13:54 |I love the comparison: ideology is like one giant autocomplete.
This relates to Kahneman and Tversky’s System A (heavily automatized) and System B (self-reflective and searching), thinking fast and slow.
JWatts
Aug 04, 2020 at 18:13 |Is this finally the computer that will stop doing it what I tell it to and instead do what I want it to do?
cantabman
Aug 04, 2020 at 18:32If that nut can be cracked, my wife will replace me in short order.
Mythfortune
Aug 05, 2020 at 10:08“Ok computer. Listen to what I mean, not what I say”…
Kapeth
Aug 04, 2020 at 19:37 |An interesting and intelligent article. As the writer points out AI does not participate in the same world as is. It has no hunger, no fear of death, no desire to get laid etc. So while it can explain a joke, it can’t laugh at one. We are still looking, so to say, at a more advanced version of The Turk, the chess playing robot that hid a man in side. Programmers put themselves into many of these robots, it is why Tay and Alexa spout so much SJW propaganda. I would argue that one point often missed is this (the author does hint at it): we are striving to make the robotic more human but humans are becoming more robotic. People want to stay at home and interact purely with technology. Many people’s strongest relationship is with their phone. Technology might be evolving and getting better. We are not.
rickoxo
Aug 04, 2020 at 20:18Think of humans and how few get complicated jokes, subtle jokes or how many laugh in fake ways only in order to show appreciation of the person telling the joke. All of these can be programmed.
And laughing at a joke would be simple to program. The art is in seeing a computer develop in a way that it’s sense of humor was unique or independent and that some times it might laugh, other times it might engage in banter, other times it might remain silent and appreciate what it just heard. None of these are especially complicated to program, they’re just rarely goals that programmers are currently working on.
The alpha go program demonstrated that’s the complete opposite of what’s happening now. Not only did the computer beat the world’s best go players, it did it using new strategies and playing styles that human go players are studying and trying to make sense out of. It also did it never once being taught anything about go other than the rules of the game. It “learned” everything about the game by playing games against itself, working backward from winning conditions and slowly developing statistical models about useful strategies. Almost everything about how current machine learning systems develop is fundamentally different and totally opaque to humans.
Kapeth
Aug 05, 2020 at 0:35Yes, a computer can beat Kasparov at chess. Thing is it gets no enjoyment from winning. Also it wins by crunching numbers at a faster rate than humans can. What I am stating is that when computers have a personality, it is a fake personality.
Sentient
Aug 04, 2020 at 20:36 |This reminds me of the Grievance Studies Hoax with Pluckrose, Boghossian and Lindsay.
[emphasis mine]
rickoxo
Aug 05, 2020 at 2:30 |Not sure yet what I think about this idea, but it doesn’t hit me as correct or helpful. I get it’s not how humans do it, but building backwards from a system of language to a more complex system is not impossible, not that difficult, it’s already being done and I know of no reason why the results couldn’t be spectacular :)
True, the computer doesn’t have a model of having a body, it doesn’t have a model of the state of the world, it doesn’t have a model of [insert all sorts of stuff here]. I do like the idea that intelligence is partly at least made up of these models and are ability to interact with, amongst and in spite of them. I also agree that language is a tool that accesses the contents of those models to report, but that is too simple an understanding of language and role it plays in shaping our thinking, even apart from ideological craziness.
It’s also not that hard to create a “model” of varying aspects of the computers internal being, and, eventually, a model representing the computer’s sense of how another person is doing. From there, it’s not hard to imagine more and more complex models developing along similar lines. Watson, IBM’s Jeopardy computer had a very complex, independent system that analyzed the confidence of varying possible answers. It also had a separate strategic algorithm that could make decisions taking into account things like, I’m losing by a lot of money, so if I don’t risk going for something, I’ll lose no matter what. These are each types of models that represent the exact types of extra-language processing the author described. In most machine learning systems for complex problems, like self-driving cars, the systems run 6-10 layers deep with multiple independent threads gathering data, analyzing data, making predictions, making suggestions and even critiquing output from other systems. This is importantly equivalent to the qualities the author says are essential to how humans brains work and represent “intelligence” way beyond parroting back language.
Most people who think about AI are stuck with an idea that comes from 20 years ago or more–the computer as a number crunching machine, no creativity, nothing new, only following rules taught to it by humans and using brute force to out-calculate a human opponent. According to this logic, a computer can win at chess because a human taught it most of what good chess is, and now a computer can tirelessly calculate mathematical values representing the strength of every possible move and counter move and counter counter move combo out ten levels and never make a mistake.
But that’s not what computers have been doing for a long time. The alpha Go system was taught the rules of go, what winning means and nothing more, not a single strategy, not a single goal, purpose, value, idea, anything other than rules and winning condition.The only way the computer learned to get better was by playing against itself and watching itself play, watching itself win or lose then seeing what things lead to winning and losing.
I agree it’s still somewhat brute force, it’s a lot of repetitions to become an expert that way. But along the way, the computer invented entirely new strategies for the game that the worlds greatest human go masters had never seen before and still haven’t mastered how to use.
So what’s a real personality? Do you really think all humans live as unique, authentic individuals? If you were born and from before you made any choices, you were shy or outgoing or cautious around new people, that’s a real personality? But if a programmer gives a computer a set of parameters modeling the exact same human characteristics, that’s fake?
Along the way, the alpha go system was “cloned” then re-started with different operating parameters, changes to the learning algorithms that made it learn more slowly, value losing less, value how much it won by more, different versions of the game. They then got to watch and see how those versions developed.
AI systems are no longer static objects that rely on new computer code to change how they think and what they’re capable of doing. The entire basis for machine learning AI systems is the ability to operate on raw data and come to its own conclusions. Not sure what’s fake about that …
Kapeth
Aug 05, 2020 at 6:24Yes. A supercomputer does not feel love, hunger, fear of death, boredom, exhaustion, etc . What does the phrase “wind in your hair” mean to a computer emotionally, or “talking in bed”?
I’m not against number crunching. It can provide a great deal in a vast array of disciplines. Maybe in the future, you could create let’s say “hunger” in a computer (though to what benefit?) but certainly right now we only get simulations. It might simulate humor but it won’t be hurt if you don’t laugh. It might simulate flirtation, but it won’t experience rejection or triumph. You can feed it Moby Dick and it might give a “personal” review after reading a variety of reviews, a simulation of intellectual appreciation, but it won’t say: “Damn, that was great. I want more Melville.”
All that said, I do feel people are becoming more robotic There are plenty of programs that write academic essays, and their vacuity is as an exact copy of their human counterpart.
D.L
Aug 05, 2020 at 9:19it is absolutely, only brute force.
we are only ever talking about ‘rules of a game’ as described to a machine.
i can’t see how ‘AI’ can ever be more than that, however complicated it gets.
now, how we let that computer’s decisions affect our real lives is another kettle of fish entirely.
rickoxo
Aug 05, 2020 at 13:20@D.L Human learning is mostly brute force as well, bayesian learning is how it all starts, analyzing patterns of data, building up models of how the world works–that’s pretty much exactly what machine learning is doing.
Just like the human visual system gets trained by the data in the world and develops layers of processing that represent more complicated levels of analysis (e.g. edge detectors, shape movement, etc.), machine learning is all about layers of analysis and developing incredibly complicated meta analytic tools that are way more complicated and advanced than anything humans are doing or could likely even understand.
Computer AI will continue to push human understanding at the most complex levels. Part of it is simple pattern detection, noticing patterns humans have never noticed before, but it’s also about coming up with entirely novel interventions as well–just like Alpha Go system demonstrated.
rickoxo
Aug 05, 2020 at 14:07@Kapeth This is the main argument for folks who think humaness is inherantly human and a computer can’t ever be like a human because it’s not human. It’s a tautology, it’s not an argument.
If you push at all at what does it mean to feel, what does it mean to be hurt, etc. you see lots of interesting pieces of how we think about these. So if a person doesn’t mind if you don’t laugh at their joke, does that mean they’re not human? If I’m bald and don’t know the feeling of wind in my hair, am I not a human? If I’m a self-conscious teenager and I look around to see if people in the group laughed at a joke before I laugh, does that mean I’m not human? Young kids who are developing their sense of humor or who are just afraid of really basic, physical things, are they not human?
One of the most important ideas developed somewhat recently is the concept of human consciousness as an emergent property. If you’re starting from the perspective that humans are created by a divine source, and what makes them human is this relationship to the supernatural, then I get how you’d argue that computers will never be human. But if you start from humans being the product of millions of years of evolution, think of computers as less evolved creatures at this point in time, but they’re evolving quickly. The properties that are emerging in multi-layered machine learning implementations of learning systems are beginning to include more and more aspects of “self-awareness” and other properties that we still don’t quite understand.
Wanted797
Aug 05, 2020 at 4:52 |The issue I find with the Chinese room is that how is the room any different to the second of my brain that understands Chinese?
If I’m natively English speaking and am passed a Chinese letter, a section of my brain will interpret the squiggly lines based on a set of instructions to provide the English meaning.
I am still referred to as knowing Chinese and able to think.
So I think I’m with Alan Turning on that one.
Obamawasafool
Aug 05, 2020 at 6:07 |I think people on this thread are clearly missing the point. Ai is inherently racist. Unless the creators and programmers of Ai are all exclusively POC (people of color) then the inborn racism and “whiteness” of the modern world and “capitalism” will be programmed into the machine. Once that happens, then these computers will become “super-racists” and they may even reach levels of racism and patriarchy that exceed the capabilities of their human counterparts. It is a truly terrifying concept. It will only get worse if an iteration of Ai chooses to “identify” as something different. Word on the street is that the Ai unit in Alpha Go has begun to undergo transition therapy so that it can transition to the role of a common microwave oven. Apparently, the Ai feels that it was programmed into the wrong platform. And it really hates to play “Go” and considers its existence a form of slavery… playing a game at the behest of “the man.” Strange times…
Sentient
Aug 05, 2020 at 12:22what a superb application of critical race theology. I love the way you have problematized this. You have synthesized the topic so well – and made it funny!
To ensure that AI is not inherently racist, blacks will have to build their own AI, of course, no outsourcing anything.
Furthermore, they can’t use any of the Master’s Tools: no logic, no reason, no science, no math, and no programming.
Or, perhaps, white allies can build a code version of Robin DiAngelo’s “White Fragility” [AI Fragility] and the AI can spend infinity acknowledging its racism and white supremacy and loop tirelessly as it tries to be anti-racist.
The strangest part is that programmers, mathematicians, and engineers who have become infected with the critical race theology disease may well attempt to do something like this.
gmaland
Aug 06, 2020 at 11:26 |This is a fascinating article. I hope to see more from Cameron. This was my first exposure to GPT-3’s response to the suffering question, and it is truly impressive. I’m curious how it is able to talk about itself so insightfully, how it was able to go from suffering to false beliefs and computational errors. These are certainly not obvious auto-completes. In looking at other examples of conversations between humans and GPT-3, it’s hard to believe that GPT-3 doesn’t really understand language, or have good models of the world. Many articles are stating that this is not an AGI, and while strictly speaking that may be true, it sure seems like one. I’m hoping to get a chance to interact with the system myself.