I have read books by both Silver and Taleb in the last couple years. Both authors have significantly changed my thinking about the world.
The Signal and the Silence looks at Silver and Taleb. It is a good review. My only real question is about the opening section, where the fate of the 2011 Boston Red Sox is described as “the worst collapse in Major League Baseball history.” I still have not recovered from the 1969 Chicago Cubs.
MathBabe looked at JPMorgan suicide is 3rd mysterious death in weeks. The result is a clear debunking using simple mathematics: JP Morgan suicides and the clustering illusion
Apparently nobody at the New York Post thought about making this simple calculation.
This story has received a lot of coverage in the last few days, even extending to Jay Leno’s monologue. Understandably, there is a lot of concern about the civil liberties implications. But there is also another question, answered by Corey Chivers: How likely is the NSA PRISM program to catch a terrorist?
We don’t really know anything about how PRISM works (NSA = Never Say Anything), but with some plausible assumptions we can estimate the answer. Suppose
Using Bayes’ rule, Chivers shows that only 1 in 10,102 of the people flagged as suspects will actually be a terrorist!
Continue reading
From Bayes: How one equation changed the way I think
Bayes’ Rule is a simple formula that tells you how to weigh evidence and change your beliefs. I don’t go around plugging numbers into a formula all the time, but nevertheless, becoming familiar with Bayes has shifted the way I think in some important ways.
After posting about The Prosecutor’s Fallacy I recalled a similar case with the Defense in the O.J. Simpson trial. The issue was summarized in What is your favorite problem for an introduction to probability?:
… one of Simpson’s lawyers, Alan Dershowitz, noted that even though Simpson beat
his wife, that hardly mattered, because in the United States, four million women are
battered every year by their male partners, yet only one in 2,500 is ultimately
murdered by her partner (1 in 1000), so, by the ‘reasonable doubt’ criterion, this is
irrelevant. The jury found that argument persuasive, but it’s spurious. The relevant
question was what percentage of all battered women who are murdered are killed by
their abusers, which ain’t 1 in 1000, but rather 9 in 10.
For a clear explanation of the details see Chances Are, by Steven Strogatz, which is reprinted in his excellent book, The Joy of x: A Guided Tour of Math, from One to Infinity.
Bayes’ Theorem … A Simple Example
Notation: Prob(A) means “the probability of event A” and Prob(A|B) is “the probability of event A, given that event B has happened.”
Bayes’ Theorem: Prob(A|B)xProb(B) = Prob(B|A)xProb(A)
Now, Prob(A|B) and Prob(B|A) are often confused by even the most intelligent of people. The confusion often appears in legal cases and is sometimes called the Prosecutor’s Fallacy. Bayes’ Theorem relates these two distinct conditional probabilities.
Followed by a straightforward example of why this really matters.
Back in 1976, when I got an M.S. in Statistics from Stanford, the dominant interpretation of probability and statistics was the Frequentist view. The alternative Bayesian interpretation was definitely a minority position.
In recent decades the Bayesian view has been gaining ground, especially after the spectacular success of one of its practioners, Nate Silver, in predicting the results of the 2012 U.S. Presidential election. Silver has written an excellent book, The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t, about forecasting. He gives some vivid examples of Bayesian methods.
The main point of Silver’s book is quite clear in the title: Real world data is full of noise. All too often people see some random fluctuation in the data and think that it represents some real pattern. Silver gives examples from many fields, including sports, the stock market, earthquakes, politics, and economics, that show this. In other cases, e.g. weather forecasting and climate change, there is a discernable signal in all of the noise. Silver neatly debunks some of the bad statistical methods used by the deniers of global warning.
Another good book about Bayesian probability is From Cosmos to Chaos: The Science of Unpredictability, by by Peter Coles. Coles assumes a little more comfort with mathematical notation than Silver, but the actual arguments do not require more than algebra. While discussing the history of probability theory from its roots in gambling, he concentrates on physics and astronomy, which also contributed significantly to the development of statistics. He is a strong advocate of Bayesian probability and suggests the Bayesian view avoids some nasty issues in the interpretation of statistical mechanics and quantum mechanics, notably that in
the latter subject there is no reason for the Many Worlds Interpretation. Incidentally, he has also argued that the conventional interpretation of Sherlock Holmes is wrong. See The Return of the Inductive Detective.
The Frequentists vs. Bayesian debate has also made Xkcd. The implication is that some level we are all Bayesians, even if we don’t admit it.
On an issue in reasoning with probabilities, Ethan Siegel discusses the Inverse gambler’s fallacy in The Last Refuge of a Science-Denying Scoundrel.
From Hilbert Space to Dilbert Space, and beyond 