I am running a multiple regression analyses on a sample of 1800 respondents. The dependent variable is the mean of a 5-point likert scale and I have 6 predictors (antecedents) also using mean of 5-point likert scales.
I have three different groups in my dataset and I am mainly interested in HOW they differ from oneanother rather than IF. Research question: How do the effects of antecedents of Organizational Identification differ among different stakeholder groups?
To compare the groups I have made dummy variables for the three groups. One has N=1000 where the others have N=400 (more or less)
I want to compare the importance of the predictors/antecedents per stakeholder group. For this I have made interaction variables (e.g. Predictor1 * Group1)
I have put all this (6 antecedents, 3 groups and 18 interactions and the control variables) in the regression analyses. The biggest group is used as reference so in the model there’s 2 groups and 12 interactions visible.
Now if I understand correctly, the b-value of one of the interaction predictors is the extra effect relative to the reference dummy interaction. So if b-value for a predictor moderated by group 2 is .34 and group 1 is the reference then I could state that the effect is stronger in group 2 than in group 1. The p-value shows the significance.
However, as I have a large dataset it is quite sensitive and although the interactions show fewer significant results, the main effects are significant with p-values like .0000000000000453
This causes me concern about the interpretation and I would like to focus on effect sizes independent of sample size. I have read several papers stressing the importance of effect size and the overratedness of p values.
So in this case, what would be the best approach to describe the real significance (practical importance) of the differences between the groups?
I have been reading a lot and so far I have come across:
- Cohen’s B – effect size in differences between means – but can you use them for differences in b-values in multiple regression?
- Calculate a r value from the t-value belonging to the b-value and it’s SE. Using sqrt(t^2 / t^2 + df)
- Cohen's F2 – but this seems some sort of invert of R2 and seems to apply to the model, not differences between groups. For example, I have an R2 for the overall model of .51 but what does this say about the size of the effect of the differences?
Extra confusion about applicability is caused by that I believe to have learned that the interpretation of dummies in regression is limited. Also from trying to figure it out from similar questions on here I have heard some say you can't assess effect sizes of interactions individually – so how else would I best go about it regarding my reseach question?
I am still reading up but the deadline of my thesis is due soon and I have been struggling with this for days so I have come here to seek help. Any help is welcome, Thank you.
Best Answer
1) Please tell me what software you are using.
2) If the interaction is insignificant you should remove it from your model and re-run the analysis.
3) If there is still a significant interaction, then you don't need to interpret the main effects.
4) I usually don't dummy code, and just put variables in as factors. So if you end up having a significant interaction between a categorical variable and continuous variable, then you should plot the groups at different levels of the continuous variable.
5) I found this article to be helpful regarding effect sizes: http://journal.frontiersin.org/article/10.3389/fpsyg.2013.00863/abstract
And download the Excel sheet that is talked about within the article, it's a handy resource.