We now test \(H_{0j} : \mu_j = 0\) for \(j = 1,\ldots,100\). We compute the \(100\) p-values, and then construct a vector recording whether the \(j\)th p-value is less than or equal to \(0.05\), in which case we reject \(H_{0j}\) , or greater than \(0.05\), in which case we do not reject \(H_{0j}\), for \(j = 1,\ldots,100\).

p.values <- rep(0, 100)
for (i in 1:100){
    p.values[i] <- t.test(x[, i], mu = 0)$p.value
}
decision <- rep("Do not reject H0", 100)
decision[p.values <= .05] <- "Reject H0"

Since this is a simulated data set, we can create a \(2 \times 2\) table similar to Table 13.2.

> table(decision, c(rep("H0 is False", 50), rep("H0 is True", 50)))
                  
decision           H0 is False H0 is True
  Do not reject H0          40         47
  Reject H0                 10          3

Therefore, at level \(\alpha = 0.05\), we reject just 10 of the 50 false null hypotheses, and we incorrectly reject 3 of the true null hypotheses. Using the notation from Section 13.3, we have \(W = 40\), \(U = 47\), \(S = 10\), and \(V = 3\). Note that the rows and columns of this table are reversed relative to Table 13.2. We have set \(\alpha = 0.05\), which means that we expect to reject around \(5\%\) of the true null hypotheses. This is in line with the \(2 \times 2\) table above, which indicates that we rejected \(V = 3\) of the 50 true null hypotheses (\(6\%\)).

Questions

• MC1: In the simulation above, for the false null hypotheses, the ratio of the mean to the standard deviation was only \(0.5/1 = 0.5\). This amounts to quite a weak signal, and it resulted in a high number of Type II errors. If we instead simulate data with a stronger signal, so that the ratio of the mean to the standard deviation for the false null hypotheses equals \(1\) (e.g. \(1/1\)), what would you expect to happen?
  1. We would make more Type II errors.
  2. We would make less Type II errors.
  3. Changing the signal strength has no impact on the expected number of Type II errors.
• You can validate your answer with the code below.