I have noticed that the sign at the front of the log-likelihood ratio formula changes depending on what source I have been looking at.
Below are the two formula I have seen, exactly as they were written (in case the notation is offering something I am missing):
$LLR = -2 ( LL_{\rm Simple} – LL_{\rm General})$
(Source)
or
$LLR = 2(LL_{\rm new} – LL_{\rm baseline}$)
(Source: Discovering Statistics Using SPSS, By Andy Field)
I was wondering if someone could tell me the significance of the sign at the front of the model? My suspicion is that it is something to do with whether you are introducing or removing parameters.
Thanks.
Best Answer
The real answer is so short as to essentially require only a comment, but I hate to leave more questions without an answer when there's an obvious one.
There's no mystery here, this is simple arithmetic. When you swap the order of the terms inside the parentheses, you invert the sign. The two forms are algebraically identical.
When the more general model is second, the sign in front will be negative, as in your first equation. If you swap it so that it is first, then you have multiplied the terms inside the parentheses in the first equation by -1, so there's a compensating negative sign comes out the front to still leave the whole thing positive.
\begin{eqnarray} LLR &=& -2\, ( LL_{Simple} - LL_{General})\\ &=& -2 \times(-1\times-1)\times\, ( LL_{Simple} - LL_{General})\\ &=& (-2 \times -1)\times\, ( (-1\times) LL_{Simple} - (-1\times) LL_{General})\\ &=& 2\, ( - LL_{Simple} + LL_{General})\\ &=& 2\, (LL_{General} - LL_{Simple}) \end{eqnarray}
Note that in the second equation you gave, the baseline model is the simpler one, so it's of the form at the end of my (trivial) derivation above.
I should mention that I don't usually remember* the right order for the first form (or equivalently, I don't remember which one goes on the numerator in the likelihood ratio), but it's trivial to work out, since only the likelihood for the more general model can be higher, and the form of the log-LR we want will be positive. Since there's a -2 out the front, the more general model goes second (or on the denominator in the ordinary LR).
* nor will I be likely to ever expend any effort memorizing it, since it's easy to figure out what is required when I need to. I actively avoid allocating mental effort to remembering things I know can figure out as I go. It'll probably eventually get committed to memory on its own, if I need it often enough to make a difference.