We are currently investigating time trends in mental health. We are questioning our interpretation of our analyses using confidence intervals and wonder if someone could advise us.
We use repeated cross-sectional data (8 time points) and examine whether an independent variable explains the trend in mental health problems (the outcome). To do this, we first examine whether the independent variable is significantly associated with mental health problems. Then we enter the ‘time’ or ‘cohort’ variable as a dummy variable (with the first year as the reference) and investigate how much of the time trend in mental health (i.e., the differences in mental health problems between the reference and the other time points) is explained by the independent variable. To do this, we compare the regression coefficients of time on mental health problems before and after entering the independent variable.
A limitation of this approach is that the comparison of the regression coefficients before and after entering the independent variable is not backed up by any statistical procedure. We are currently considering overcoming this limitation by comparing the 95% confidence intervals of the regression coefficients before and after entering the independent variable, using the rule of thumb that about 25% of the overlap of 95% CIs corresponds to a statistically significant difference at the p=.05 level. However, most discussions regarding the overlap of 95% CIs deal with the mean scores of a dependent variable between two groups. This engenders two questions that make us hesitant to apply this method to our model. Specifically:
first, can we indicate a significant difference in the magnitudes of two regression coefficients based on the overlap of their 95% CIs?
second, if the answer to the above is yes, is it possible to compare the 95% CIs of two regression coefficients that are obtained from two different (nested) regression models?
(Adding an exempleary case)
Suppose that we compare grade 9 students’ mental health problems (categorical variable) in year 2010 vs. in year 2015. The level of students’ mental health problems is higher in 2015 than in 2010. A working theory is that a higher level of school anxiety in year 2015 compared to year 2010 is what contributed to the increase in mental health problems.
To test this, first, we entered a cohort variable (0=Year 2010 / 1=Year 2015) to see the difference in mental health problems between students in 2010 and those in 2015. The results show that the cohort variable presented a significant odds ratio of 2.0.
Second, we added school anxiety (continuous variable) to the above model. School anxiety presented a significant odds ratio of 1.5. In addition, the magnitude in odds ratio of the cohort variable decreased from 2.0 (in the first model) to 1.5 (in the second model).
We wondered if there is any way statistically to support that the odds ratio of the cohort variable in the first model (2.0) is significantly different from that in the second model (1.5). The above questions in the original post was to ask whether it would be correct to observe the overlap of the 95% CIs between them. But any advice would be greatly appreciated on any method to enable us statistically to test if school anxiety contributed to the decrease in the magnitude in the odds ratio of the cohort variable. Our actual data is more complicated in that we have 8 time periods/cohorts.
Best Answer
You fit two models: $\beta_0 + \beta_1X_1$ and $\beta_0^`+ \beta_1^`X_1 +\beta_2^`X_2$, where $X_1$ = cohort, $X_2$ = school anxiety. You want to compare $\beta_1$ and $\beta_1^`$.
The condition that $\beta_1 = \beta_1^`$ is $X_2$, school anxiety, has no effect on the outcome or $X_2$ is orthogonal with $X_1$. Generally, $X_2$ is not orthogonal with $X_1$. So you just check if $X_2$ is significant or not. If not, go back to the first model and OR = 2 is corrected estimate. If yes, work with the second model, and OR = 1.5 is correct one.