Solved – Acceptable values for variance, aic and bic in multilevel models

aicmixed modelmultilevel-analysisstata

I'm building a multilevel model from a sample of 820 observations at level 1 and 11 groups (level 2). I'm using stata xtmixed.

Running the empty model (including just the dependent variable) I get:

variance at level 1 (residual) = 0.6
variance at level 2 (_cons): 0.03
So the Intraclass correlation coefficient is 5%
AIC: 1860
BIC: 1875

Are my absolute values of variance too small? Too close to zero? Is there a recommended threshold given the sample size?
From my previous question I learnt that having a very low ICC is ok.

Are AIC or BIC values too high, considering the sample size and the variance values? I mean, is there a rule that given around 800 observations tells me which should be an appropriate starting value for AIC/BIC?

Best Answer

Mine was a trivial question. Thanks to John I learnt that absolute values for variance depend on the unit of measure (and magnitude) of the dependent variable (I'm just testing the empty model right now).

Also, AIC and BIC depend on the number of observations. If I randomly drop half of the observations, AIC and BIC decrease of about 50%.

Related Solutions

Solved – How to one empirically demonstrate in R which cross-validation methods the AIC and BIC are equivalent to

In an attempt to partially answer my own question, I read Wikipedia's description of leave-one-out cross validation

involves using a single observation from the original sample as the validation data, and the remaining observations as the training data. This is repeated such that each observation in the sample is used once as the validation data.

In R code, I suspect that that would mean something like this...

resid <- rep(NA, Nobs) 
for (lcv in 1:Nobs)
    {
        data.loo <- data[-lcv,] #drop the data point that will be used for validation
        loo.model <- lm(y ~ a+b,data=data.loo) #construct a model without that data point
            resid[lcv] <- data[lcv,"y"] - (coef(loo.model)[1] + coef(loo.model)[2]*data[lcv,"a"]+coef(loo.model)[3]*data[lcv,"b"]) #compare the observed value to the value predicted by the loo model for each possible observation, and store that value
    }

... is supposed to yield values in resid that is related to the AIC. In practice the sum of squared residuals from each iteration of the LOO loop detailed above is a good predictor of the AIC for the notable.seeds, r^2 = .9776. But, elsewhere a contributor suggested that LOO should be asymptotically equivalent to the AIC (at least for linear models), so I'm a little disappointed that r^2 isn't closer to 1. Obviously this isn't really an answer - more like additional code to try to encourage someone to try to provide a better answer.

Addendum: Since AIC and BIC for models of fixed sample size only vary by a constant, the correlation of BIC to squared residuals is the same as the correaltion of AIC to squared residuals, so the approach I took above appears to be fruitless.

Solved – ICC reduced to 0 in a multi-level model. Should this be suspicious

This seems pretty unsurprising to me.

In your model, the ICC can be interpreted as the proportion of the variance not explained by covariates that is due to variation between classes.

If instead of using "proportion of fully prejudiced respondents in the class", you used "average level of prejudice within each class" as a covariate, this would certainly reduce the ICC to zero, as this covariate explains all the variation between classes by definition.

So if "proportion of fully prejudiced respondents in the class" is strongly correlated with "average level of prejudice within each class", as you might well expect, it's not surprising that it also reduces the ICC to zero.

Best Answer

Related Solutions

Solved – How to one empirically demonstrate in R which cross-validation methods the AIC and BIC are equivalent to

Solved – ICC reduced to 0 in a multi-level model. Should this be suspicious

Related Question