Section 40 Post hoc analysis: Multiple Testing

40.1 Multiple Testing of Hypotheses

• Multiple comparisons of means allows pairwise comparison of means and identify if two means are statistically significantly different.

• One can assess the statistical significance of differences between means using a set of confidence intervals, a set of hypothesis tests or both.

• When testing multiple null hypotheses by comparing the multiple means across the levels of a factor, use of multiple t tests will inflate the probability of declaring a statistically significant difference when the null hypothesis is true.

• In other words, multiple testing of null hypotheses increases the type 1 error.

• It is very important to keep this into account while making multiple comparisons because the chances of committing a type I error for a series of comparisons is greater than the error rate for any one comparison alone.