Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence.

I enjoyed parts of this book, but overall found it quite a chore to get through. Just a few years on, some things have changed so much that parts of the book feel dated already. I enjoyed catalog of the problems, approaches and dangers, but found some of the discussion of them pedantic.

Just a few annotations:

“A big breakthrough in artificial intelligence. It seems somewhat likely that it will happen sometime in this century, but we don't know for sure”.

-I don't think so, but I truly hope to see that with my eyes.

“The idea of a coming technological singularity has been popularized, starting with Vernor Vinge's seminal essay and continuing with the writings of Ray Kurzweil and others. The term ‘Singularity,' however, has been used confusedly in many disparate senses and has accreted an unholy aura of techno-utopian connotations”.

-Agree.

“Machines matching humans in general intelligence have been expected since the invention of computers in the 1940s”

-Add another century to it.

“The fact that the best performance at one time is attained through a complicated mechanism does not mean that no simple mechanism could do the job as well or better. It might simply be that nobody has yet found the simpler alternative”.

-That's my bet for developing a SI.

“How far are we currently from achieving a human whole brain emulation?. One recent assesment presented a technical roadmap and concluded that the prerequisite capabilities might be available around mid-century, though with a large uncertainty level”.

- I also doubt a WBE in the near future. Probably in 2120?.

“Final goal: ‘make us smile'.
Perverse instantiation: ‘paralyze human facial musculatures into constant beaming smiles”.

- I laughed on this because it makes sense.

“...These observations make it plausible that any type of entity that developed a much greater than human level intelligence would be potentially extremely powerful. Such entities could accumulate content much faster than us and invent new technologies on a much shorter timescale. They could also use their intelligence to strategize more effectively than we can”.

- I liked this book a lot specially because of the Realistic Pessimism about an SI. I consider there must be a real concern to think about that level of intelligence.

Should have been 1/5th the length. Reads like someone spouting hypothetical scenarios off the top of their head. The only reason it gets a 2 stars is because it isn't “The Circle” by Dave Eggers.