It’s also an open thread (and a SLOW night in the ER)

There has been a debate in the world of Statistics on what to do now that there are such large N studies when the math is set up for much much smaller N’s.

Please read the details here

The most intriguing fix has been here with us for 350 years. The Reverend Thomas Bayes put forth this:

 is a conditional probability: the likelihood of event  occurring given that  is true.

 is also a conditional probability: the likelihood of event  occurring given that  is true.

 and  are the probabilities of observing  and  independently of each other; this is known as the marginal probability.

This is the simplest example:

Drug testing[edit]

Tree diagram illustrating drug testing example. U, Ū, “+” and “−” are the events representing user, non-user, positive result and negative result. Percentages in parentheses are calculated.

Suppose that a test for using a particular drug is 99% sensitive and 99% specific. That is, the test will produce 99% true positive results for drug users and 99% true negative results for non-drug users. Suppose that 0.5% of people are users of the drug. What is the probability that a randomly selected individual with a positive test is a user?

33.2%

Even if an individual tests positive, it is more likely that they do not use the drug than that they do. Why? Even though the test appears to be highly accurate, the number of non-users is large compared to the number of users. The number of false positives outweighs the number of true positives.

To use concrete numbers, if 1000 individuals are tested, there are expected to be 995 non-users and 5 users. From the 995 non-users, 0.01 × 995 ≃ 10 false positives are expected. From the 5 users, 0.99 × 5 ≈ 5 true positives are expected. Out of 15 positive results, only 5, about 33%, are genuine.

This illustrates the importance of base rates. Daniel Kahneman has argued that the formation of policy can be egregiously misguided if base rates are neglected when using statistics as a basis for guiding public policy.^[3]

The importance of specificity in this example can be seen by calculating that even if sensitivity is raised to 100% and specificity remains at 99% then the probability of the person being a drug user only rises from 33.2% to 33.4%, but if the sensitivity is held at 99% and the specificity is increased to 99.5% then the probability of the person being a drug user rises to about 49.9%.

As you can see, without Bayes, and Bayesian probability interpretation, we lose how a subjective degree of belief should rationally change to account for availability of related evidence. Bayesian inference is needed for these much larger N studies and for ever more complicated statistical modeling especially if policy or important outcomes are determined by these date sets.