I have a regression with two continuous predictors and one dichotomous predictor in Model 1 and two interactions of each of the continuous predictors with the dichotomous predictor in Model 2. The coefficient for one of the interaction terms is significant. However, the F test is not significant (neither for Model 1 nor for Model 2). Should I still interpret that significant interaction or should I just assume that neither the predictors, nor the interaction terms are useful in predicting the outcome variable?
Solved – Not-significant F but a significant coefficient in multiple linear regression
f-testmultiple regression
Related Solutions
ANCOVA and multiple regression are mathematically identical. In matrix algebra terms, both are $Y + XB + e$. If you don't have much variance explained in ANCOVA, you won't in regression.
The main problem I can see with your colleague's approach is that of inflating type I error by running multiple tests.
Statistical significance is, generally, of less importance than many people think it is. From a scientific/substantive point of view, if the effect sizes are small they are usually not interesting (although there are exceptions.... If you could reduce the number of, say, plane crashes by a small amount, airlines would probably be interested).
@Robert Kubrick's answer is correct as far as it goes. Beyond testing for statistical significance, you will want to assess the size of any "interaction" effect.* Here, it's probably debatable whether the difference in slopes is great enough to matter in a practical sense. For your audience you will want to quantify the extent to which {the change in y for each unit change in x} differs depending on presence/absence of z. Armed with that, they'll be better informed as they make up their own mind how much of an interaction there is. The t and p statistics don't supply that information.
You wrote,
A quick plot of the data appears to show that there is a larger effect of z when x is not present, but that presence of z still contributes significantly even when x is present.
This would be true if you exchanged each instance of x for z and verse-vice-a.
*"You may find such effects described using the terms moderator effect, product effect, joint effect, or multiplicative effect. [...] Distinguish true interactions, which only apply in experimental studies, from the types of joint effects seen in most research, including observational, correlational, and descriptive studies. In many of these cases, the two variables which are said to "interact" are really part and parcel of rather than orthogonal to one another. Thus any investigation into joint effects is best done with careful attention to construct validity and the nature of the variables measured." From YellowBrickStats.com.
Best Answer
It sounds like your F-tests are not the ones you want, the default F-tests returned by most software compare the fitted model to a null (intercept-only) model. It sounds like you want to do an F-test comparing Model 2 to Model 1.
If you do this, you could still find that the F-test comparing these two models is insignificant, while the t-test on one of the interactions is significant. It is important to keep in mind that these are different tests, answering slightly different questions. The t-test on the interaction term is a test of how strongly the data reject the hypothesis that the interaction term is zero, holding fixed all the other coefficients in the extended model, including the other added interaction term. The F-test is a test of how strongly the data reject the hypothesis that all the added terms are zero.
If the two terms you added in the extended model have one t-stat that is narrowly significant and another that is insignificant, the F-test can be insignificant. One way to think about why this can happen is that adding multiple terms to a model creates a multiple testing problem. The chances of spuriously finding a large coefficient on one of the interaction terms increases as you add more terms. The F-test takes this into account, while the individual t-tests for each term do not.