Solved – Relative variable importance values vs. magnitude of effect

aiclme4-nlmemixed modelmodel selection

I have ran a series of models to see which best fit the response variable and I got the following (for the model average of all models with a $\Delta AIC < 2$). I am currently learning models so this output is one of my first:

         Estimate Std. Error z value Pr(>|z|)    
(Intercept)   102.7190     5.5300  18.575  < 2e-16 ***
HDr            -1.5495     0.3451   4.490 7.11e-06 ***
MF.vs.OF2      -7.6780     3.7507   2.047  0.04065 *  
NHDp           -0.5145     0.2909   1.769  0.07695 .  
NHDr           -1.4164     0.4663   3.037  0.00239 ** 
Site2           6.1477     2.7400   2.244  0.02485 *  
tide.h.l2      -7.2546     2.6914   2.695  0.00703 ** 
tide.inc.out2  -5.8486     2.6187   2.233  0.02553 *  
HDp            -0.3773     0.2732   1.381  0.16731    
mean.for.rate  -0.3966     0.3220   1.232  0.21807    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Full model-averaged coefficients (with shrinkage): 
 (Intercept)        HDr  MF.vs.OF2       NHDp       NHDr      Site2  tide.h.l2 tide.inc.out2        HDp
  102.718962  -1.549499  -5.734171  -0.239550  -1.416373   5.336532  -7.254627     -5.848553  -0.044795
 mean.for.rate
     -0.081734

Relative variable importance:
  (Intercept)           Age           HDp           HDr mean.for.rate      MF.vs.OF          NHDp          NHDr 
         1.00          0.00          0.12          1.00          0.21          0.75          0.47          1.00 
         Site      tide.h.l  tide.inc.out 
         0.87          1.00          1.00

I have seen the estimates described as the effect of the variable and this is discussed in results sections as an important value to report (in regards to the size of them and their direction (+ve/-ve)). (the paper I was reading was stating that those with the bigger or smaller numbers had the greatest effect, even quoting that one was 48% lower than the other). However if this is what is reported and discussed, why would the relative variable importance vary in relation to the estimate? It seems that this should also be looked at but am not sure how the z and p values are calculated from a model.

Therefore, I would like to know which is more important when trying to discuss the findings. I admit that my knowledge is limited, but I would like to grasp this in simple terms if I could.

Also, I used a dredge command to find all the model combinations from a global model and found seven models (out of 1024) had a $\Delta AIC < 2$. I thought that this was a good cut-off for averaging, but could technically use any delta value as the cutoff.

Thank you.

Best Answer

It is a little unclear what you are asking in your question but it seems you are confusing relative variable importance and the magnitude of an effect. I am providing a link to some good literature sources that discuss the use of AIC in the ecological disciplines.

Relative Variable Importance

Burnham and Anderson (2002) describe a simple way to quantify variable importance.

Page 168: Estimates of the relative importance of predictor variables $x_j$ can best be made by summing the $AIC$ weights across all the models in the set where variable $j$ occurs.

The larger the sum of model weights, the more important a variable is relative to the other variables. The relative importance values can be used to rank the variables. However, to use this method, one must have an equal number of models for each variable; otherwise, some variables will be over represented or under represented resulting in biased relative importance values.

Burnham and Anderson (2002), Page 169: When assessing the relative importance of variables using sums of the $AIC$ weights, it is important to achieve a balance in the number of models that contain each variable $j$.

Related Solutions

Solved – Understanding what a factor is in a model

If you ask lmer not to estimate an intercept, it's smart enough to realize this means you must want the absolute, rather than relative, factor estimates (some of the output below has been removed to save space). You should probably center the dependent variable before running the model without an intercept, as is done in the code below.

> library(lme4)
> 
> # Construct sample data
> x <- as.factor(rep(c("A","B","C"), 10))
> z <- as.factor(rep(c("D","E","F","G","H"),6))
> y <- rnorm(30, as.numeric(x))
> 
> 
> # Run model with intercept: gives relative effects
> summary(lmer(y~x+(1|z)))
Linear mixed model fit by REML 
Formula: y ~ x + (1 | z) 
   AIC   BIC logLik deviance REMLdev
 92.76 99.77 -41.38    81.12   82.76
Random effects:
 Groups   Name        Variance Std.Dev.
 z        (Intercept) 0.00000  0.00000 
 Residual             0.97188  0.98584 
Number of obs: 30, groups: z, 5

Fixed effects:
            Estimate Std. Error t value
(Intercept)   1.5572     0.3118   4.995
xB            0.4971     0.4409   1.128
xC            1.1541     0.4409   2.618

> # Run model w/o intercept: gives absolute effects
> # First center y so no confounding of intercept with effects
> y <- scale(y, scale=FALSE)
> summary(lmer(y~x+(1|z)-1))
Linear mixed model fit by REML 
Formula: y ~ x + (1 | z) - 1 
   AIC   BIC logLik deviance REMLdev
 92.76 99.77 -41.38    81.12   82.76
Random effects:
 Groups   Name        Variance Std.Dev.
 z        (Intercept) 0.00000  0.00000 
 Residual             0.97188  0.98584 
Number of obs: 30, groups: z, 5

Fixed effects:
   Estimate Std. Error t value
xA -0.55042    0.31175  -1.766
xB -0.05329    0.31175  -0.171
xC  0.60371    0.31175   1.937

Otherwise the factor estimates have to be relative (although not necessarily relative to the first factor), as with an intercept present, the factors and intercept together are perfectly multicollinear.

Solved – Relative variable importance with AIC

This is some further advise/discussion I was given:

AIC RIW can only be calculated from a balanced candidate model set. If you have 3 variables (e.g. repro, time & WR) then the balanced set (without interactions) is

repro
time
WR
repro + time
repro + WR
time + WR
repro + time + WR
intercept only

the number of models in the set is 2 to the power of the number of explanatory variables (in this case = 8) with 2-way interactions your candidate model set ALSO includes the following (i.e. in addition to those above)

repro + time + repro*time
repro + WR + repro*WR
time + WR + time*WR
repro + time + WR + repro*time
repro + time + WR + repro*WR
repro + time + WR + time*WR

If you want the 3-way interaction, then you would ALSO add this to all of the models described above.

Each variable relative importance weight is then the SUM of ALL AIC-weights from models that contain that variable. Because AIC-weights are standardized to sum to one within a candidate model set, then RIW for each variable can range from 0 to 1.

Do not divide the result by the number of models it is contained in – it is the total sum. I would only use these for balanced candidate model sets; I wouldn’t use RIW for a smaller number of models.

NOTE that if you include interactions, then you can only compare the RIWs of main effects with each other, and you can only compare the RIWs of interactions with each other. You cannot compare main effect RIWs with interaction RIWs (because main effects are present in more models than interactions).

FYI: a strong explanatory variable will have a RIW of around 0.9, moderate effects of around 0.6-0.9, very weak effects of around 0.5-0.6 and below that, forget about it. For interactions, a strong effect could be >0.7, moderate >0.5. If you’re not using RIWs then simply look at your model table and see if you get consistent improvements in AIC when you add specific variables, and by how much. Strong effects will often give you improvements in AIC of >5, moderate 2-5 and weak 0-2. If you don’t get an improvement at all, then it isn’t explaining anything.

if you don’t have a balanced candidate set, but DO have the AIC weights (which it appears you do), then you can simply use the ratios of these to determine the strength of support for one model over another. E.g. if you have model 1 with AIC-weight of 0.7 and model 2 with an AIC-weight of 0.15; then model 1 has 4.6 times more support from the data than model 2 (0.7/0.15). You can use this to assess the relative strength of variables as they go in and out of models. But you don’t NEED to do these calculations – and can simply refer the reader to the table. Especially if you have a dominant model; or a series of models at the top that all contain a particular variable. Then it is simply obvious to everyone that it is important.

Best Answer

Related Solutions

Solved – Understanding what a factor is in a model

Solved – Relative variable importance with AIC

Related Question