PSTM: Chinese Thought Experiment

The Idea: Let me tell you about a man sitting in a room who doesn’t speak Chinese. I know, weird way to start an email.

John Searle has perhaps the most well-known thought experiment relating to AI called the Chinese Room Argument. It goes like this: Imagine a person in a room. You can use yourself to make it more vivid. Now, you don’t speak Chinese. However, there’s a book in there that has all the Chinese symbols in all sorts of combinations. Moreover, it has suggested replies based on the order of Chinese symbols, in Chinese too.

Someone outside the room who can’t see in is sending in Chinese messages. You see them, match the book, and send out the response that is suggested.

People outside the room marvel at how brilliant you are at speaking Chinese. Inside the room, nothing remotely resembling understanding has occurred.

This experiment was popularized in the 1980s, and was a fairly damning comment on the deterministic nature of AI of the day.

However, I want to take what has been happening with LLMs and turn the thought experiment around, because I think it is worth considering.

Imagine you have the same setup. We’ll use English this time, because I want a different language, one I know you understand (and if you don't speak English, how'd you end up here in the email?)

Now, imagine you have hired a new research attendant, and you are starting to regret it, because the new research attendant has a weird verbal tic that is getting on your nerves.

Instead of differentiating between human and computer, this new research attendant is insistent on referring to the underlying mechanisms at play when any phenomenon happens. Weird, but stay with me. You have a room, some English goes in, and intelligible English comes out. This research attendant comes back and tells you the result of the experiment.

“We had some input. It was received by a bunch of neurons as info, and it actually progressed through a system of interconnected neurons in fascinating and complex ways we don’t understand. From what we do understand, this input was parsed into a semantic meaning, data from the past was looked at, compared to the meaning, and the most reasonable response was chosen and given back.”

Are we talking about a human or a machine?

The Advice: Pay attention to what is happening in AI. I’m getting bullish. I think it’s scary, and our risks of planetary extermination are non-zero (never a good thing) but I also think a lot of the fear and fearmongering is overblown.

You also want to get a handle on it so that you choose how to interact with it, and don’t let that choice get made for you.