8 LMM: Model Selection

8.1 Model selection: Random effects

\[H_0: \sigma^2 = 0\]

\[H_1: \sigma^2 > 0\]

You may use the likelihood ratio test (LRT) to test the hypotheses.
Fit two models with and without the specified random effect term and fit both models using REML or ML. Note you should not change the fixed effect terms.
The test statistic is calculated by subtracting the −2 REML log-likelihood value for the Model with and without the specified random effect (the nested model).
The asymptotic null distribution of the test statistic is a mixture of \(\chi^2\) distributions, with 0 and 1 degrees of freedom, and equal weights of 0.5.
A convention is to divide the p-value by 2 because we are testing the null hypothesis that the variance of the random effect equals zero, which is on the boundary of the parameter space for the variance.
SPSS does not provide a direct way to compare the random effect term using the LRT. You have to do it manually if you wish so.
We will not investigate this further in this course. We will rather chose our random effect terms for the model using other criteria like convergence issues, estimates, confidence interval of variance etc.

\[H_0: \beta_1 = 0\]

\[H_1: \beta_1 \ne 0\]

Likelihood Ratio Test (LRT)
- The LRT statistics are constructed by taking the differences of the -2 Log likelihood of two nested models.
- It follows a chi-squared distribution with degrees of freedom equal to the difference in the number of parameters of the models.
- To use the likelihood ratio test to test fixed effects, the model must use maximum likelihood method to estimate the variance.
- The p-values generated by the LRT for fixed effects are approximate and tend to be small.
- SPSS does not provide a direct way to conduct LRT; you have to do it manually or using script.
- We will not investigate LRT further here in this course.
Type 3 sum of squares and corresponding F-statistic for the primary research objective is useful; it provides the global significance of a fixed term
If the global F-statistic is significant, you can also evaluate the p-value from the t-statistic of the model outputs. This is similar to the approach we take for a standard analysis of variance or regression model.
Information criterion:
- Useful and recommended for model selection particularly when the models are not nested
- The smaller the information criterion, the better is the model
- Akaike’s Information Criterion (AIC)
- Schwarz’s Bayesian Criterion (BIC)
- Hurvich and Tsai’s Criterion (AICC)
- Bozdogan’s Criterion (CAIC)

The summary estimates with t-statistics and corresponding p-values are guidelines but when data are unbalanced, these may not be informative
Check the performance statistics like logLikelihood, AIC, BIC for different models
Model with smaller AIC and BIC values are parsimonious and should be taken forward
We can formally compare the model using the likelihood ration test (LRT), however, be careful while conducting the LRT as it assumes that the model is fitted on the same data
You must fit the model using the maximum likelihood (ML) method for conducting model selection
The REML-based likelihood for models with different fixed effect structure are not comparable
Always check the convergence of the model; if the convergence is doubtful, you should not do any model comparison. For example, comparison between a model that converged and another complex model that did not converge.
Note SPSS may still provide complete outputs even when the model did not fulfill the given convergence criteria.
Even if the model does not converge, the model outputs tell us the reasons why the model did not converge. In most scenarios, this could be due to the over-specification of the fixed or random effect structure of the model that is not supported by the data. It suggests that we should simplify the model.