Machines Like Us interviews: John Searle

Thanks for the excellent interview.

I was glad you followed up on the answers a few times, but they still did not persuade me. I think it's rather obvious that in the Chinese room not the man passes the Turing test as Searle proposed, but the executed symbols do pass it, as the same symbols certainly work in any room with a machine that executes them the same way as the man did.

But the biggest problem for me was in his next answer - to quote: "But the putative Chinese speaker inside me has the same problem. By following the steps of the program, he can give Chinese answers to Chinese questions, but has no way to attach any meaning to any of the symbols." Nearly correct - except the sentence should continue with: "unless the symbols he follows are symbols to attach meaning to symbols." Which is just the premise of AI of which he tried to get rid of here. It's still in his system and he couldn't exorcise it. He showed that you can (maybe) build a Chinese speaking system which does not understand. But he does not show the other way round - that all software systems will fail to understand.

What he misses in my opinion is that the execution of symbols itself does matter. He concentrates only on the symbols. To take your book example - the book certainly isn't important, but only the substrate of the story. But that is still not enough - only through reading the story gets an existence. A book which no one reads might contain a story, but it's only different from all potentially existing but newer written stories insofar that it is easier accessible to readers. Note that we talk of stories - not of books as for the books existence it's certainly a difference if it is written or not. Well, the story certainly got executed already once on writing, but that's beside the point here. Symbols create no existence *unless they are executed*. A few examples to show the difference:

First lets copy the Chinese room. We now have 2 Chinese rooms with a man in each of them. Both man still have the same symbols. But this time the speaker asks them different questions. And suddenly context matters - because despite the man and the symbols still being the same we get two different answers. And while we can replace as well the man as we can replace the medium on which the symbols are written, we can't remove the part of the execution as little as we can replace the rules by any random rules. So both things are important - and not just the symbols alone.

Another example are computer games. You can't play the sourcecode. Only through execution you get the ghosts to hunt your Pacman. And while the symbols of the sourcecode does define the statespace of every hunting possibility in this game, that is still just an infinite number of states. But only those states that actually got executed (by playing) do really get into existence.

An executed symbol is not the same as a symbol. And such a system created of executed symbols is from within as real as our world is for us. And this is not just a matter of states. At least not a matter of the states within such a system as two systems can have identical states and still have independent existences (it can be explained by states when embracing the system running those systems, like our world having processors).

Now to get to the question of meaning - how much meaning does a story actually need in it's target system to be understood? How much meaning does a target system need to understand Chinese? What does it mean to steer Pacman?

The story itself defines how large the understanding system has to be. If it's not understood then the target rule system is insufficient. It is possible to execute symbols which parse the story without understanding it the way it was meant to be understood - and so far all software systems do so. But certainly from that one cannot infer that therefore sufficiently complex realities to contain entities able to understand stories can't be run on computers.

The rules of the Chinese room might or might not understand Chinese. The man does not - but as shown above he doesn't really matter as he isn't the one passing the Turing test here. The executed rulesystem does pass it and nothing in this experiment demonstrates or dismisses the idea that a Turing test is sufficient to detect understanding. It only shows that the Turing test *might* be passed by a non-understanding system. But the rule system might as well get conscious when run on the man-machine.

And what with Pacman? Doesn't our control of it with our amusement and frustration differ from a software bot steering it? Obviously not within the computer game. It won't notice the difference between a human or a second independent software playing it as long as it gets the same inputs. And this second independent software system might or might not amuse and frustrate itself while controlling Pacman.

And last - what's with the Pizza? Well, the Pizza is not the same because it is not executed in the same system. It gets it's very existence by being executed by the physic rules of our world. The simulation can use a cookbook and get a Pizza in it's own system which is within that system just as tasty. But as soon as you run that same Pizzacode on another identical simulation the Pizza in there - even if it was copied from the original Simulation - has an independent existence which is as far from the one in the first system as from the one in our world.

Each Pizza-eating event has an independent existing context and I hope soon AI's will enjoy that event as much as I do.

Great interview! Thank you both for a stimulating read.

> “Does anybody really believe that if we had done a computer simulation of the digestion of pizza on a computer that we can then rush out, buy a pizza, and stuff it into the computer and the machine will digest a pizza? Of course not.”

No, of course not. But that misses the point and is unfair to the argument. The model of the digestive system is in virtual space but Professor Searle’s pizza exists in our universe. Of course a virtual digestive system can’t eat a physical pizza. But if it were the right kind of model then it WOULD digest a virtual pizza. And virtual rain would make virtual people wet.

Stephen Harnad once got cross at me regarding my own views on simulation and said that he’d believe me only when a virtual airplane could carry him to Seattle. Again, that’s unfair. A virtual airplane only has to be able to fly a virtual Stephen Harnad to a virtual Seattle. After all, a physical airplane can’t carry a virtual person to a virtual place!

In my view, Searle is making a mistake by thinking of the computer as simply being “programmed to behave like” a digestive system or whatever. It’s possibly true that in principle we could emulate an entire digestive system, a weather system or a brain by DIRECTLY implementing it as a set of rules for symbol manipulations (“if this happens then do that”). It’s even possible that we could program it in advance to behave “as if” it were digesting pizza (or even to answer “I think therefore I am” when asked whether it is conscious). Indeed this is the way that early AI scientists tended to work, and I think Searle was absolutely right to criticize it. But that isn’t the type of model Paul is talking about in this interview, and in a way Professor Searle is falling foul of the very computational functionalism he objects to, by presuming that we could ever successfully program a direct (zombie) simulation of general intelligent behavior in this way (or even the translation of Chinese).

I think we need to differentiate between first-order and higher-order simulations. Searle is quite right that a model is not the real thing. If you write equations that approximate the outward behavior of atoms, then you haven’t made real atoms. Nevertheless, if those imitation atoms spontaneously combine to form systems that behave like molecules, the molecules spontaneously form structures, and the structures start behaving like cells, say, then I would argue that these higher-order phenomena are equivalent to their real-world counterparts, even though they arise from an imitation substrate.

The mistake, IMHO, is to assume that physical things have some kind of special ontological status – that an electron is a little lump of matter sitting in space and somehow more “real” than, say, a cloud or a whirlpool or the operation of a program. All “things”, all “solid objects” are really emergent processes – an electron is a resonant state in the electromagnetic field. So if a simulated electromagnetic field spontaneously curls up into a similar resonant state (as it should if it is a good model) then I think this resulting virtual electron has just as much right to be called “real” as a so-called physical electron does. The properties of the virtual electron are not “programmed in” – they emerge from interactions that are the same as those in the physical world, even though the properties of the medium from which they arise are sham. They just exist in a different universe, that’s all.

“Syntax is not semantics”, for sure. But virtual worlds can create their own semantics. If we programmed a computer to behave like subatomic particles, and those particles were allowed to self-organize into galaxies, planets, etc. then we could legitimately expect life to arise on some of those planets. That life would exist because it is able to replicate, metabolize and compete, just like so-called real life. It’s not fake metabolism – it’s an inevitable and unavoidable consequence of the acting out of the same laws that control energy flow in our universe, even though they’re fake at the lowest level. If you put some virtual pizza into the simulation, virtual bacteria would eat it and thrive. The symbols in the underlying computer simulation of atomic theory have no semantics, but for these virtual creatures life would have real meaning. If they had brains, their own mental symbols would be fully grounded.

Unlike the pizza, it may be possible for thought and meaning to cross from one universe to another. An intelligent artificial creature that evolved (or was designed) in such a virtual environment can digest the remains of other creatures that it has cooked into a pizza, and it can think about where it is going to get its next meal. It couldn’t eat a pizza that was cooked in our universe, but its thoughts could interact with our universe. In its own world it gets its information through virtual senses, but we could perform an operation on it, remove its brain, place it in a suitable vat, made from virtual materials, and then by adding some “cheat code” to our underlying model we could connect it up instead to the sensory and motor systems of a physical robot. Now its meaning comes from our world instead. It is conscious of our world and believes it suddenly lives here.

Indeed, we could do it the other way around: we could in principle wire our own brain’s inputs and outputs to sensors located inside the computer model, whereupon we’d become conscious of the virtual world and believe ourselves to be inside it. We would still claim to be conscious and feel like we were interacting temporarily in a meaningful way with objects in that world, even if we couldn’t actually get stronger by eating them. This fact alone doesn’t suddenly stop us from being conscious, even if it renders certain aspects of our behavior (such as the search for food) temporarily meaningless. So if we claim to be still conscious while temporarily interacting with the virtual world, and a denizen of that world claims to be still conscious while interacting with ours, what’s the difference?

> “It is possible in principle to build a machine that behaves exactly like a human being but has no consciousness or intentionality. No mental life at all. Indeed, in a small way we are making fragments of such machines”

But only in a small way. As I understand it, Professor Searle’s argument for the Chinese Room presumes that it would in fact be possible to write down the rules for translating Chinese. Without such codification in a book or computer, the illustration doesn’t work. But is it really possible to translate one language flawlessly into another without knowledge of the world beyond linguistics? And wouldn't that knowledge have to be grounded in personal meaning? What would a book or program that could translate Chinese really look like? Could it simply be a set of IF/THEN rules, in which the necessary knowledge has been explicitly embedded by an intelligent designer? Or would it have to look more like a simulation of a brain, embedded in a simulation of a human-like world, in order to understand the meaning of words well enough to translate them? Suppose you asked it (in Chinese), “Does your father like pizza?” Any pre-programmed answer (in English) could be followed up by more searching questions, so the “book” would thus have to pass the Turing Test to perform flawlessly as a translator. Syntax can’t be isolated from semantics.

I’d like to suggest that Strong AI is not the hopeless goal: WEAK AI is. Weak AI presumes that we can explicitly encode the outward behavior of an intelligent being without regard to the way such behavior emerges in nature. It is functionalist in the extreme. Philosophically it holds the more comfortable position of not needing to believe that the systems are “really” intelligent or acting with “real” intentionality. But on the other hand it asserts that intelligent-like behavior can successfully be implemented in non-biology-like ways. Up to a point it can – we can vaguely translate between languages using an expert system; we can usually track the movement of a face using a vision algorithm; we can do arithmetic using a calculator – but after fifty years we’ve seen that such abstract representations don’t generalize, don’t combine well, and suffer from severe complexity explosions.

In principle it may seem possible to make a zombie – a system that isn’t conscious but always looks as if it is – but I’m not convinced this is true, at least not in polynomial time. The rules for such a system’s behavior multiply rapidly – so rapidly in fact that it’s pointless to believe that a serial digital computer of finite size could contain and compute all those rules in the time between the Big Bang and the Big Crunch. So we must of course be wary about extrapolating philosophy from what digital computation could do In PRINCIPLE. What kind of computational representation COULD do it? Well, we know that a physical human being, embedded in a physical human society, eating physical pizza can do this in real time. Admittedly it has to be conscious to do it, but as Searle says, a human being is a computational system. Now, I’m arguing that a computer simulation of atomic theory could, in principle, generate virtual intelligent beings embedded in their own society, surviving on virtual pizza. Obviously there are far too many atoms in the universe for such a simulation to work in practice either. However, a simulation like this scales well, for various reasons, while more explicit and shallow rules for outward behavior scale very badly. Sometimes it’s better, instead of trying to build something that quacks like a duck, moves like a duck and ****s like a duck, to actually build a duck. Even if it’s a virtual duck. Many of the desired effects then emerge for free. At some level of representation, I think it is practically possible to model the brain and end up with the emergence of genuine intelligent (learned) behavior, without explicitly representing it inside the program. Whether the result is REALLY conscious and shows intentionality may still be a moot point, but as long as it works I’m cool.

I think the Chinese Room argument was a valid objection to the early pioneers in AI, with their explicit, highly functionalist, symbolic representations and their lack of insistence on embodiment and situatedness. However, I don’t think it works for systems in which behavior emerges in a self-organized, systemic fashion, based strongly on the ways in which the same behavior emerges in nature. For some of us at least, the paradigm of digital computation has moved on from its declarative and procedural roots, to a pseudo-parallel, ultra-object-oriented view that even Turing and von Neumann would have found hard to believe lies hidden inside their creation. Computers aren’t just machines for doing things, they’re also places in which to build things, and although it’s still just symbol processing underneath, so is the human mind fundamentally just atoms bumping into each other. Who would have believed that atomic collisions were capable of creating consciousness?

Many people, especially in robotics, intuitively understand that the so-called Systems Reply refutation of the Chinese Room Argument (CRA) is essentially correct. Namely, it is erroneous to focus one's intuitions solely on the person's understanding--or lack thereof--rather than upon the entirety of the room's processing, including the room's interactions with external inputs (experiences) and its memory (for learning, via what can be recorded in the room, not in the person's own memory). (Note that putting the room-system inside the person's head--without changing their role in the system--changes nothing.)

But we can do better than just relying on such intuitions. The essential aspects of Systems Reply can be formalized. The person is a Universal Turing Machine (UTM). The room-system constitutes a different Turing Machine (TM) (for which the book/program is the blueprint). Thus, the person's UTM computation is totally distinct from the room's TM computation. Moreover, UTM computations are a subset of all TM computations. Therefore, the CRA omits most computations of potential interest!

So that should end it. This is not based on intutition. It's based on math. From the theory of computation, it's obvious.

Thus, the CRA fails to refute the "computer metaphor of mind", or, more properly, it only refutes a strawman version of it: "the mind works like a UTM". That is clearly false. It is also uninteresting, except within the CRA's historical context: the CRA was mainly intended to demonstrate the flaw of the "symbolic AI" paradigm that dominated the late 1970s. The CRA totally fails to refute the proposition that mind is a computation of some kind producible by some subclass of Turing Machines.

This argument, along with arguments refuting the "Godelian arguments" (cf. Roger Penrose et al.), are presented in detail in the following paper: Turing Machines And Semantic Symbol Processing: Why Real Computers Don't Mind Chinese Emperors, PDF version:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.7645
or online version (lacking figures, references, and some math symbols):
http://lyceumphilosophy.com/?q=node/30