Running a crude model using Bayesian inference, I get some results > 1 (ie, more than 100% "certain") for some combinations of "evidence". For instance, for one bit of evidence the conditional probability of the null hypothesis is 0.85 while the marginal probability is 0.77. If the prior probability is 0.9, the computed posterior probability is 1.008. Which maybe could be ascribed to rounding error, except that the next bit of evidence raises the posterior probability to 1.34.
It stands to reason that, for a problem with two hypotheses, one would have a conditional probability less than the marginal probability and the other would be greater. So the resulting P(E|H) / P(E) multiplier would be > 1. So it's hard to see how such > 1 results can be avoided in the general case.
Is this just the way Bayesian inference works, or do I likely have an error somewhere in my calcs?
Data
One "evidence":
Total 6134 samples
Total actual positives 2845
Total actual negatives 3289
Test was true 1623 times
Test was true 465 times when the "gold standard" was positive
Test was true 1158 times when the "gold standard" was negative
conditional probability of positive hypothesis = 465/2845 = 0.1634
conditional probability of negative hypothesis = 1158/3289 = 0.3521
marginal probability of test being true = 1623/6134 = 0.2646
Bayes multiplier for the negative hypothesis = 0.3521 / 0.2646 = 1.3307
As can be seen, the multiplier is significantly > 1, and with several such tests back-to-back it seems hard to avoid probabilities > 1. (Of course, I suppose one can argue that the tests aren't truly independent, and that puts the fly in the ointment.)
Fudging
So does anyone have any suggestions as to how to "fudge" non-independent measurements to improve an estimate? For the two most egregious cases I can come up with a fair estimate of how connected the measurements are, but I don't have a feel for how to factor that knowledge in.
Best Answer
You're making a mistake somewhere.
Yes, $P(E|H)$ can be greater than $P(E)$, but $P(E|H)P(H)$ should never be greater than $P(E)$, since
$$ P(E) = P(E|H)P(H) + P(E|\ {\rm not} \ H)P( \ {\rm not} \ H). $$
EDIT: Presumably the "evidence" is the uncertain test, and the "hypothesis" is the result of the "gold standard test". As the other listed probabilities appear to be right (though they're misleading named; I would instead list them as "conditional probability of positive test result given (positive) negative hypothesis), the marginal hypothesis probabilities (which you don't list) must be wrong, and indeed the Bayes multiplier is right, and greater than one. (But it's a bad test where a positive result increases your belief of a negative hypothesis.)
Initially $P(H = t)$ is just the fraction of gold-standard tests that are true, 2845/6134. Similarly, $P(H = f) = 3289 / 6134$.
$$ \begin{align*} P(H = f | E = t ) &= P(E = f | H = t) P(H = f) / P(E = t) \\ &= (1158/3289)*(3289/6134)/(1623/6134) \\ &= 1158/1623 \end{align*} $$
As should be expected, when you know the test returns true, you get out the fraction of true tests that are revealed by the gold standard test to be negative.
You'll have to be careful when trying to iterate Bayes rule. The next "$P(H | E)$" needs to be "$P(H | E_1, E_2)$", because you're interested in the chances after collecting both bits of data. It's not that hard to setup the formulas, but you have to be extremely careful about what your symbols mean. Jaynes recommended always writing probabilities as probabilities conditioned on "background information". Adding this conditioning to the standard formula may help you see how the formula must be used when iterating.