Suppose You Are Testing A Hypothesis H0 And Youraba
W ORK 1 Suppose You Are Testing A Hypothesis H0 And Yourba
Suppose you are testing a hypothesis H0, and your background assumptions X imply that the only alternative hypothesis to H0 is H1. Assume that initially you are indifferent between H0 and H1. You plan to reject H0 if a certain test statistic falls into a rejection region. Let R be the proposition "The test statistic falls in the rejection region." You are given that P[R|H0X] = 0.05, which is the probability of rejecting H0 when it is true (Type I error rate). Additionally, assume that P[R|H1X] = 0.04, the probability of rejecting H0 when H1 is true (power of the test).
The question asks: What is the correct posterior probability P[H0|RX]? Here, R is the event that the test statistic falls in the rejection region. Using Bayes' theorem, the posterior probability of H0 given that R has occurred is calculated as:
P[H0|RX] = (P[R|H0X] * P[H0]) / P[R|X]
where P[R|X] is the total probability of R under the background assumptions X, given by:
P[R|X] = P[R|H0X] P[H0] + P[R|H1X] P[H1]
Since you are initially indifferent between H0 and H1, the prior probabilities are both 0.5, i.e., P[H0] = P[H1] = 0.5. Plugging in the known values:
- P[R|H0X] = 0.05
- P[R|H1X] = 0.04
- P[H0] = P[H1] = 0.5
Calculating P[R|X]:
P[R|X] = (0.05)(0.5) + (0.04)(0.5) = 0.025 + 0.02 = 0.045
Now, the posterior probability P[H0|RX] is:
P[H0|RX] = (0.05)(0.5) / 0.045 = 0.025 / 0.045 ≈ 0.5556
This shows that, even though the prior belief was indifference, observing R slightly increases the posterior belief that H0 is true to approximately 55.56%. The fact that P[R|H1X] is less than P[R|H0X] indicates that the test is actually more likely to reject H0 when H1 is true, but the posterior still favors H0 given R, due to prior odds and the likelihoods.
A real-world example of such a scenario could involve quality control testing where a slightly higher false alarm rate (Type I error) happens in conjunction with a very low detection rate when the alternative is true. For instance, a drug test might have a false positive rate of 5%, but a true positive rate of only 4%. Such a situation might be encountered in screening for a rare disease with similar test characteristics. The posterior belief after observing a positive test would be used to decide subsequent steps, emphasizing the importance of understanding how the likelihoods and prior beliefs combine.
Consideration of Optional Stopping and Different Experimental Setups
The second part of the problem introduces an “optional stopping” scenario, where Adam performs a series of independent trials, each resulting in either a Good (G) or Bad (B) outcome. The data D is specified as: “The number of trials was 12 and the number of bad results was 3”. The hypotheses are:
- H0: The success rate (probability of a good result) is 50%
- H1: The success rate is 75%
The prior probability assignment for H0 is 10%, hence P[H0] = 0.10, and P[H1] = 0.90. The questions ask to compute the posterior probability of H0 given the data under two assumptions about the experimental process.
Part a: Fixed Total Number of Trials
Assuming the total number of trials is fixed at 12 regardless of the outcomes, the data follow a binomial distribution. The likelihoods are:
- P[D|H0] = Binomial probability of observing 3 bad results out of 12 trials with success probability 0.5.
- P[D|H1] = Binomial probability of observing 3 bad results out of 12 trials with success probability 0.75.
Calculating these likelihoods:
P[D|H0] = C(12,3) (0.5)^3 (0.5)^9 = 220 (0.5)^12 ≈ 220 0.00024414 ≈ 0.0537
P[D|H1] = C(12,3) (0.75)^3 (0.25)^9 ≈ 220 0.422 0.0000038 ≈ 220 * 0.0000016 ≈ 0.00035
The total probability of the data P[D] using Bayes' theorem:
P[H0|D] = (P[D|H0] P[H0]) / (P[D|H0] P[H0] + P[D|H1] * P[H1])
Substituting the values:
P[H0|D] = (0.0537 0.10) / (0.0537 0.10 + 0.00035 * 0.90) = 0.00537 / (0.00537 + 0.000315) ≈ 0.00537 / 0.005685 ≈ 0.944
Thus, the posterior probability that the success rate is 50% given the data is approximately 94.4%.
Part b: Sequential Experiment with Stopping After 3 Bad Results
In this scenario, Adam continues trials until he obtains 3 bad results. Since we observe exactly 3 bad results in total, the likelihood now follows a negative binomial distribution, which models the number of trials until a fixed number of failures occur.
The probability of observing exactly 3 bad results in 12 trials, with the total number of trials being a random variable, is given by the negative binomial distribution:
P(n) = C(n-1, r-1) p^r (1 - p)^{n - r}
where n is the total number of trials, r = 3 (number of bad results), p is the success probability, and the trials are independent.
Under hypothesis H0 (p=0.5):
P(n=12|H0) = C(11, 2) (0.5)^3 (0.5)^9 = 55 0.125 0.001953 ≈ 0.0134
Under hypothesis H1 (p=0.75):
P(n=12|H1) = C(11, 2) (0.75)^3 (0.25)^9 ≈ 55 0.422 0.0000038 ≈ 55 * 0.0000016 ≈ 0.00009
The prior probabilities remain the same, P[H0]=0.10, P[H1]=0.90.
Applying Bayes' theorem:
P[H0|D] = (0.0134 0.10) / (0.0134 0.10 + 0.00009 * 0.90) ≈ 0.00134 / (0.00134 + 0.000081) ≈ 0.00134 / 0.001421 ≈ 0.943
The result confirms a high posterior probability favoring H0. The difference in stopping rules affects the likelihood calculations substantially, but the outcome remains similar in this case.
Part c: Reflection on the Results
The surprising aspect of these outcomes is that modifying the experimental stopping rule does not significantly alter the posterior probability in this scenario. Despite stopping after a fixed number of bad results, the calculated posterior remains heavily in favor of H0. This underscores the importance of the likelihood function form in Bayesian updating. The negative binomial approach effectively tailors the likelihood to the stopping procedure, yet the data still heavily supports H0 due to the considerably higher likelihood ratio favoring H0 under the observed data.
These insights highlight critical considerations in statistical hypothesis testing and sequential analysis. It demonstrates that different experimental designs, especially choices about when to stop collecting data, can influence Bayesian inference and the interpretation of evidence. Recognizing such nuances aids in designing robust experiments and interpreting their results properly within a Bayesian framework.