Plan

Frivillig: Mette trenger hjelp med å sette opp en liste over interaktiv grafikk statistikkgruppa skal lage til ulike emner i statistikk. Vi skal lage dem ved å bruke R og noe som heter Shiny. Her er en oversikt over noe som er laget ved caltech:

https://statistics.calpoly.edu/shiny

Hva savner du i ST1201? Mette vil gjerne høre!

Ressurser:

Oppgave 1: Multiple testing

In genome-wide association studies the aim is to test if there is an association between a genetic marker and a trait. This means that an hypothesis test is performed for each marker. We have data from \(1000\) markers and for each marker, \(j=1,\ldots,1000\), we perform an hypothesis test: \[ H_0: \mu_j=0 \text{ vs. } H_1: \mu_j\neq 0\] where \(\mu_j=0\) means that there is no association between the marker and trait. From hypothesis test \(i\) we calculate a \(p\)-value \(p_i\) (based on some continuous test statistic). \(P\)-values are described below:

library(knitr)
pvalues=scan("https://www.math.ntnu.no/emner/TMA4267/2017v/CompEx3P2pvalues.txt")
hist(pvalues,nclass=20)

cutoffvec=c(0.05,0.05/500, 0.05/1000, 0.1/500,0.1/1000,1-(1-0.05)^(1/1000))

res=matrix(ncol=3,nrow=length(cutoffvec))
for (i in 1:length(cutoffvec))
{
  res[i,1]=sum(pvalues<cutoffvec[i])
  res[i,2]=sum(pvalues[1:500]<cutoffvec[i]) 
  res[i,3]=sum(pvalues[501:1000]<cutoffvec[i])
}  
df <- data.frame(cbind(cutoffvec,res))
names(df) <- c("cut-off", "p-values < cut-off", " first 500 p-values < cut-off","last 500 p-values < cut-off")
kable(df)
cut-off p-values < cut-off first 500 p-values < cut-off last 500 p-values < cut-off
5.00e-02 51 22 29
1.00e-04 0 0 0
5.00e-05 0 0 0
2.00e-04 0 0 0
1.00e-04 0 0 0
5.13e-05 0 0 0

a) Explain with words what is a \(p\)-value.

Assume that we reject all null-hypotheses with corresponding \(p\)-values below 0.05. How many null-hypotheses will we then reject?

What is a false positive finding? Do we know the number of false positive findings in our data?

b) Let the number of false positive findings for our data be called \(V\). What is the definition of the familywise error rate FWER?

What does it mean to “control the FWER at level 0.05”?

The Bonferroni method will control the FWER. What cut-off on \(p\)-values should we use if we want to control the FWER at level 0.1 for our data with the Bonferroni method? Call this cut-off \(\alpha_B\). How many null-hypotheses will we reject with this new cut-off?

To see the effect of choosing different cut-offs on \(p\)-value on the number of false positive findings we need to know which null hypotheses are true and which are false. Let us assume that all the 1000 \(p\)-values come from true null hypotheses. What does this imply about the number of rejection in a) and b? What if only the first 500 \(p\)-values come from true null hypotheses?

c)

The Sidak method for FWER control can be derived by assuming that the \(m\) \(p\)-values are independent. Look at the derivation in 5.2 https://www.math.ntnu.no/emner/TMA4267/2017v/multtest.pdf, and explain to each other what is happening.

Is the Sidak or the Bonferroni method the most strict method? Calculate the \(\alpha_{\text{LOC}}\) for the Sidak method for our 1000 hypothesis.

d)

Describe briefly what is meant by the following two terms: \(p\)-hacking and reproducibility crises. What is the relationship between these two terms and multiple testing.

Oppgave 2: MCQ

Type I errors

What is a commonly used name for the type I errors?

    1. true positives
    1. false positives
    1. false negatives
    1. true negatives

\(p\)-value from true null hypothesis

For a continuous test statistic that gives an exact \(p\)-value, what is the distribution the \(p\)-value when the null hypothesis is true?

    1. Normal
    1. Chisquared
    1. Exponential
    1. Uniform

FWER

\(V\)=number of false positives and \(R\)=number of rejections. The familywise error rate FWER is

    1. \(\text{E}(V/R)\)
    1. \(\text{E}(V)\)
    1. \(P(V/R>0.05)\)
    1. \(P(V>0)\)

Bonferroni

\(\alpha\)=level for control of FWER. \(\alpha_{\text{loc}}\)=cut-off on \(p\)-value, \(m=\)number of tests.

What is the Bonferroni rule?

    1. \(\alpha_{\text{LOC}}=m\alpha\)
    1. \(\alpha_{\text{LOC}}=\frac{\alpha}{m}\)
    1. \(\alpha_{\text{LOC}}=\alpha^m\)
    1. \(\alpha_{\text{LOC}}=(1-\alpha)^{1/m}\)