I have a 2(truthfulness) x 2 (immediate test) study where I am looking to compare 2 logistic regression models.
Both are classifying the same outcome variable: truthfulness (truthful versus deceptive — an experimental manipulation where they were told to tell the truth or lie). The independent variable in both is a score on a delayed test.
The first model uses scores for people who did the immediate test, the other model uses scores for people who did not do the immediate test. I want to know whether truthfulness can be classified better using scores on the delayed test when people have previously done the immediate test.
However, I am confused as to how to compare logistic regression models.
Is it possible to do that even though models are estimated from different samples? Is it possible to interpret differences between classification accuracy tables as meaningful? And is it possible to statistically test the differences between models?
Best Answer
I am confused. You say the DV is the experimental condition? This cannot be. The thing you manipulate is always an independent variable; and score on the test should be the dependent variable.
Given that you have 2x2 tables (apparently 2 of them) you could look at percent correct in each, or sensitivity and specificity in each, and compare these. Testing the difference between two proportions is standard stuff.
Are the two groups comparable? I don't know. It depends on how you got them. Are they both random samples from the same population?