Section 41 Multiple Testing: FWER & FDR

41.1 Familywise Error Rate

  • The familywise error rate (FWER) is the probability of making at least one false discovery when performing multiple hypotheses testing.

  • The familywise error rate remains the most accepted parameter for ascribing significance levels to statistical tests when we are interested in definitive results.

  • Different methods of controlling familywise error rate are:

    - Bonferroni correction

- Tukey's Honestly Significant Difference

- Holm

- Dunnett

- Sidak


41.2 False Discovery Rate

  • The false discovery rate (FDR) controls the expected proportion of false discoveries amongst the rejected hypotheses when making multiple hypotheses testing.

  • It is a less stringent condition than the family-wise error rate, so these methods are more powerful than the others.

  • If a study is viewed as exploratory, or if significant results can be easily re-tested in an independent study, control of the false discovery rate (FDR) is often preferred.

  • Different methods of controlling false discovery rate are:

    - Benjamini & Hochberg (1995)

- Benjamini & Yekutieli (2001)


41.3 Guidelines

  • Always be careful while considering multiple hypotheses testing.

  • Avoid multiple hypotheses testing, if possible.

  • Define the type 1 error while planning the experiment.

  • Conduct an Anova first and check the F-statistic.

  • Present mean and confidence interval of means.

  • Identify the comparisons of interests during the planning of the experiment.

  • Do not pick and choose after obtaining the data.

  • Identify orthogonal contrasts.

  • Identify (g-1) levels of orthogonal contrasts where g is the level of treatment group.

  • In case of multiple testing, think carefully regarding experimental hypotheses vis-a-vis a suitable approach (FWER or FDR).