What does the B stand for in Benoit B. Mandelbrot?
Fractal Humor
Leave a reply
Tag Archives: math
More on the problems of PRISM
Thomas Bayes looks through the NSA’s “Prism”
This story has received a lot of coverage in the last few days, even extending to Jay Leno’s monologue. Understandably, there is a lot of concern about the civil liberties implications. But there is also another question, answered by Corey Chivers: How likely is the NSA PRISM program to catch a terrorist?
We don’t really know anything about how PRISM works (NSA = Never Say Anything), but with some plausible assumptions we can estimate the answer. Suppose
- If a terrorist is in the system, the probability is 99% that PRISM will flag him/her.
- An good guy has only a 1/100 chance of being flagged as potential terrorist.
- The actual number of terrorists is quite small, say 1 in 1,000,000.
Using Bayes’ rule, Chivers shows that only 1 in 10,102 of the people flagged as suspects will actually be a terrorist!
Continue reading
Thomas Bayes and O.J. Simpson
After posting about The Prosecutor’s Fallacy I recalled a similar case with the Defense in the O.J. Simpson trial. The issue was summarized in What is your favorite problem for an introduction to probability?:
… one of Simpson’s lawyers, Alan Dershowitz, noted that even though Simpson beat
his wife, that hardly mattered, because in the United States, four million women are
battered every year by their male partners, yet only one in 2,500 is ultimately
murdered by her partner (1 in 1000), so, by the ‘reasonable doubt’ criterion, this is
irrelevant. The jury found that argument persuasive, but it’s spurious. The relevant
question was what percentage of all battered women who are murdered are killed by
their abusers, which ain’t 1 in 1000, but rather 9 in 10.
For a clear explanation of the details see Chances Are, by Steven Strogatz, which is reprinted in his excellent book, The Joy of x: A Guided Tour of Math, from One to Infinity.
The Prosecutor’s Fallacy
Bayes’ Theorem … A Simple Example
Notation: Prob(A) means “the probability of event A” and Prob(A|B) is “the probability of event A, given that event B has happened.”
Bayes’ Theorem: Prob(A|B)xProb(B) = Prob(B|A)xProb(A)
Now, Prob(A|B) and Prob(B|A) are often confused by even the most intelligent of people. The confusion often appears in legal cases and is sometimes called the Prosecutor’s Fallacy. Bayes’ Theorem relates these two distinct conditional probabilities.
Followed by a straightforward example of why this really matters.
Is there really pi in the sky?
Bayesian Probability
Back in 1976, when I got an M.S. in Statistics from Stanford, the dominant interpretation of probability and statistics was the Frequentist view. The alternative Bayesian interpretation was definitely a minority position.
In recent decades the Bayesian view has been gaining ground, especially after the spectacular success of one of its practioners, Nate Silver, in predicting the results of the 2012 U.S. Presidential election. Silver has written an excellent book, The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t, about forecasting. He gives some vivid examples of Bayesian methods.
The main point of Silver’s book is quite clear in the title: Real world data is full of noise. All too often people see some random fluctuation in the data and think that it represents some real pattern. Silver gives examples from many fields, including sports, the stock market, earthquakes, politics, and economics, that show this. In other cases, e.g. weather forecasting and climate change, there is a discernable signal in all of the noise. Silver neatly debunks some of the bad statistical methods used by the deniers of global warning.
Another good book about Bayesian probability is From Cosmos to Chaos: The Science of Unpredictability, by by Peter Coles. Coles assumes a little more comfort with mathematical notation than Silver, but the actual arguments do not require more than algebra. While discussing the history of probability theory from its roots in gambling, he concentrates on physics and astronomy, which also contributed significantly to the development of statistics. He is a strong advocate of Bayesian probability and suggests the Bayesian view avoids some nasty issues in the interpretation of statistical mechanics and quantum mechanics, notably that in
the latter subject there is no reason for the Many Worlds Interpretation. Incidentally, he has also argued that the conventional interpretation of Sherlock Holmes is wrong. See The Return of the Inductive Detective.
The Frequentists vs. Bayesian debate has also made Xkcd. The implication is that some level we are all Bayesians, even if we don’t admit it.
On an issue in reasoning with probabilities, Ethan Siegel discusses the Inverse gambler’s fallacy in The Last Refuge of a Science-Denying Scoundrel.
Statistics Hell
From Andy Field.
You hate statistics, you despise maths, you stain your pants at the mention of sphericity. Normal people love these things, but you, barrel of number-fearing sputum that you are, are terrified. Afraid for your worthless life you leapt pathetically to the internet for guidance. A freak spark of lightening hit your house sending a bolt of electricity through your computer. Sparks flew like tendrils from the screen, fusing with your face and sucking your head and body into the number vortex that is statistics hell. Your crime is evacuating your bowels at the mention of a t-test, your punishment is eternity in statistics hell. I am the gatekeeper, the evil ruler of this world of numbers. Although you might confuse me for a human full of empathy and compassion for those taking their first wobbly steps through this horrific world of equations, underneath my skin I am numbers without a soul.
Another winner on Tuesday
I have a degree in Statistics (M.S., Stanford, 1976). I really enjoyed reading all of this.
Why Math is Like the Honey Badger: Nate Silver Ascendant
If ever there was an iron-clad case to be made for math literacy, it’s what happened over the last few weeks with the New York Times‘ star statistician Nate Silver and his 538 blog (named after the 538 votes in the electoral college).
Nate Silver does it again! Will pundits finally accept defeat?
For the practical implications: How Conservative Media Lost to the MSM and Failed the Rank and File
Also this Tweet:
Karl Rove forms new SuperPAC to run negative ads against “the scourge of math and statistics that’s ruining America”.
Life Imitates Art
Tom Lehrer: “I’ve often thought I’d like to write a mathematics textbook someday because I have a title that I know will sell a million copies; I’m gonna call it Tropic of Calculus.” (1965)
Steven Strogatz, The Joy of x (2012)
From Hilbert Space to Dilbert Space, and beyond 