When a new technology appears, humans are very optimistic. We see the speed of transportation increase by orders of magnitude in a few decades (horses to jets) and we think this is going to continue. In 1966, Time Magazine ran a survey of futurists. “82 scientists agreed that a permanent lunar base will have been established long before A.D. 2000 and that men will have flown past Venus and landed on Mars.” It’s 2026, and we haven’t built a city in Antarctica. I cannot buy an airline ticket to go there. Getting from New York to London takes twice as long as 40 years ago on the Concorde.
The same pattern can be seen in other areas, from human longevity (know any 120-year-olds) to battery energy density. Computing/AI hasn’t seen a bottleneck yet, we seem to be in the exponential part of the curve. But why wouldn’t it? There are some obvious bottlenecks along the way that we’ll have to face. For example:
New information: So far, AI looks a lot like human intelligence but faster and scalable. This is because it trains on us. Some people believe that at some point, AI will spontaneously start thinking in a way that is radically different from ours. But why would this happen? Computers can generate synthetic data and learn from playing games against themselves, but where will new facts come from? Eventually it needs to autonomously learn from the environment.
Compute/energy. We could get to point where doubling the capacity of AI requires 100x more compute and energy. This would clash with human institutions, in fact it’s already starting.
Exploring the physical world. Thought experiments happen at the speed of thought, but real experiments happen at the speed of physics. Sometimes learning the ground truth takes years or decades. There is an ongoing Harvard study on happiness that started in 1938, for example. Self-driving cars are gathering data about some aspects of the world, very slowly. Robots might gather other kinds, but it won’t happen overnight.
The supporters of the idea that AI could become instantly superintelligent via recursive self-improvement dismiss these concerns. They assign a high probability to the fact that AI could overcome all these bottlenecks and slowdowns somehow. That it won’t run into something like “ok, for my next training run I need to do a robot sweep of the world. This is going to take 37628.4 years approximately. Launching manufacturing process...”
Maybe it’s possible that AI will somehow jump to another phase instantly and find a shortcut that we can’t imagine. But nobody has made a good argument for it other than the lazy “there is a nonzero chance this will happen.” And this is sophistry: there is an infinite number of things that have a nonzero chance of happening and we don’t worry about most of them: we could get hit by an undiscovered comet, a gamma-ray burst, a nearby supernova.
But AI doom is appealing: it grabs attention, it makes people money. It’s very hard to be intellectually fair about it, because emotions get in the way.
What complicates matters is that AI is indeed a potentially dangerous technology even if it has a negligible chance of exterminating all humans. There are immediate dangers that need to be addressed, and every AI lab has strong conflicts of interest.
So if someone wants to make a serious case that significant resources need to be spent on preventing AI doom, they need to do better than say “nonzero odds.” If someone were making the case that we need to urgently create an interplanetary government and laws because in the next decade people will be massively immigrating to other planets, the burden of proof would be pretty high. Same for building a shield against rogue comets, or an alien defense system. Why is it different for AI extinction? It can’t be because it makes for good sci-fi.
Faith in bottlenecks slowing things down is reasonable. But that doesn't tell you how long it will take - how far away from the explosion you need to standing to see those bottlenecks in operation.
No serious person doubts that every accelerating process eventually hits something in the real world that slows it down and changes its shape. But we can not know the actual limits in advance. This needs to be repeated: AI is a new process; we can be certain it will be limited, but currently, we truly don't know where or how.
When the Wright brothers flew, who predicted that the flying machines would continue to accelerate until we found the limit of that acceleration at the economics of spaceships (!) within a handful of decades? Who gave that outcome non-zero odds in 1903? 1910? 1920?
The fact that serious, technical people right now see no immediate bottlenecks must give you pause. The fact that they report increasingly high odds that the accelerating process that has given us these "flying machines" is unlikely to hit the limit of before something as weird as "spaceships" must give you pause.