The Signal and the Noise

We like future predictions to be presented to us as absolutes: this stock will rise, it will rain, this team will win. Our brains have a hard time making sense of uncertainty and margins of error. We like our realities to be oversimplified, have clear incentives for taking actions. Plus, we don't like to get wet when we leave our umbrellas at home despite being told there's a minimal chance of rain (this goes so far that the weather channels deliberately tweak what their models predicted, to avoid angry rained-on viewers, see Wet Bias).

This book is from 2012, post Moneyball and post 2008 financial crisis. Predictions and the rise of big data was on everyone's mind. Who predicted the housing bubble and who made money of it? Silver goes through different domains where predictions and probabilistic thinking play a major role: the weather, baseball, poker, chess, the stock market, the climate. He starts on how we fail at predicting the occurrence of earthquakes, despite knowing the likely frequency of their occurrences. And he ends on the statistics of terrorist attacks, whose ratio of frequency and impact (their ‘power laws') are not unlike earthquakes.

In hindsight, after the occurrence of an event, it's always easy to look back and see the signal that hinted at it. Yet, while we're in it, it's incredibly hard to distinguish which part of the data is a signal (clear indicator) and which is noise (irrelevant fluctuations, irrelevant other signals). Which part of a baseball player's results are skill and which are luck? Statistical analysis and the accumulation of large amounts of data have helped with isolating signals in some domains, but not all. We're also living in a world that's increasing its volume of information exponentially, tied together in complex systems, making it increasingly important to develop probabilistic thinking. Bayesian logic is the use of knowledge of prior events to predict future events, while continuously updating your hypothesis about the future, when new data comes in. (So far, every morning the sun has risen from the east, therefore I assume it will rise from the east tomorrow morning as well).

Great book, still highly relevant of course. The climate chapter felt a bit messy, and there potentially was a bit too much content from the author's old poker-days, but else this was a highly informative and interesting read. Good to get comfortable with the ‘known unknowns' before having to deal with ‘unknown unknowns'.