In his book The Signal and the Noise, Nate Silver presents this example application of Bayes's Theorem on pp. 247-248:
Consider a somber example: the September 11 attacks. Most of us would
have assigned almost no probability to terrorists crashing planes into
buildings in Manhattan when we woke up that morning. But we recognized
that a terror attack was an obvious possibility once the first plane hit
the World Trade Center. And we had no doubt we were being attacked
once the second tower was hit. Bayes's theorem can replicate this result.
You can view the complete example in Amazon.com's previw, and I've made the two pages available here.
Silver assumes the prior probability of a terrorist plane attack to be 1 in 20,000. After the first plane crash, using Bayes's Theorem he updates that to 38%. And after the second plane crash, he comes up with a 99.99% probability. However, I think he may be mistaken. I'll provide the details below.
To be precise, let us define the following three events:
- $PC$ = Plane Crash: At least one plane crashes into a Manhattan skyscraper on a given day.
- $TPA$ = Terrorist Plane Attack: At least one plane is intentionally crashed into a Manhattan skyscraper on a given day.
- $APC$ = Accidental Plane Crash: At least one plane is accidentally crashed into a Manhattan skyscraper on a given day.
We assume all plane crashes into buildings are either terrorist plane attacks or accidental (i.e. $PC = TPA \cup APC$). Using historical data, Silver estimates the prior probability of an accidental plane crash to be 1 in 12,500. In summary: $$P(TPA) = \frac{1}{20000},$$$$P(APC) = \frac{1}{12500}.$$
Furthermore, Silver assumes $P(APC) = P(PC|\overline{TPA})$ (which is true if $APC$ and $TPA$ are independent events).
Applying Bayes's Theorem, he comes up with
$$\begin{align}P(TPA|PC) &= \frac{P(PC|TPA) \times P(TPA)}{P(PC|TPA) \times P(TPA) + P(PC|\overline{TPA})(1-P(TPA))} \\
&= \frac{1 \times \frac{1}{20000}}{1 \times \frac{1}{20000} +
\frac{1}{12500} \times (1 – \frac{1}{20000})} = 0.385\end{align}$$
Silver continues:
The idea behind Bayes's theorem, however, is not that we update our
probability estimates just once. Instead, we do so continuously as new
evidence presents itself to us. Thus our posterior probability of a
terror attack after the first plane hit, 38 percent, becomes our
prior probability before the second one did. And if you go through the calculation again, to reflect the second plane hitting the World
Trade Center, the probability that we were under attack becomes a
near-certainty — 99.99 percent.
That is (this is Silver's calculation): $$P(TPA|PC) = \frac{1 \times 0.385}{1 \times 0.385 +
\frac{1}{12500}(1-0.385)} = 99.99 \%$$
"Cool!" I thought, until I thought a bit more. The problem is that you can apply the same logic to calculate the conditional probability of an accidental crash, too. I'll spare you the math, but I come up with $P(APC|PC) = 0.615$ after the first crash, and $P(APC|PC) = 99.997\%$ after the second.
So we can be almost certain the second plane crash is a terrorist attack, and we can be even more certain that it's accidental?
I think the problem is that when Silver applies Bayes's Theorem after the second crash, he uses the updated probability of a terrorist plane attack as his prior, but fails to update the prior probability of an accidental plane crash (which should become 0.615). After the second crash, then, the correct formula is
$$P(TPA|PC) = \frac{1 \times 0.385}{1 \times 0.385 +
0.615(1-0.385)} = 0.504$$
Similarly, the probability that we're observing an accidental crash given that there have been two crashes is
$$P(APC|PC) = \frac{1 \times 0.615}{1 \times 0.615 +
0.385(1-0.615)} = 0.806$$
Question 1: Am I correct that Nate Silver is doing it wrong?
Question 2: Am I doing it right?
Best Answer
Correct, there is no contradiction here.
If we know that the first crash was a terrorist attack, then the second crash would be more likely another terrorist attack.
The same reasoning with accidental crashes.
No. There is no need to update the rate of accidental crashes. IMHO, Nate implies that accidental crashes don't include terrorist ones. Otherwise, he couldn't multiply probabilities in the denominator.