Confirmatory Factor Analysis – CFA Fit Measure Thresholds for New Scales: Validity and Scale Assessment

confirmatory-factorscalesvalidity

A well established questionnaire measuring interpersonal relations is modified to measure interorganizational relations. 11 parameters of the relation are determined by 4 items each. Items are removed until the best possible $\alpha$ per scale is achieved (.58 – .81). This results in some scales with 3 items, some scale with 4 items.

A confirmatory factor analysis is applied to calculate the estimates of the items and verify the internal structure of the questionnaire.

As expected with some $\alpha$ as low as .58 and .61 the CFA does not fit very well. Since it is "kind of" a new scale, is it ok to use the predictions for further modelling and report them?

The fit indices are (bad):

  Number of observations                           251

  Estimator                                         ML      Robust
  Minimum Function Test Statistic             2475.330    2029.585
  Degrees of freedom                               854         854
  P-value (Chi-square)                           0.000       0.000
  Scaling correction factor                                  1.220
    for the Satorra-Bentler correction

Model test baseline model:

  Minimum Function Test Statistic             5718.372    4583.881
  Degrees of freedom                               946         946
  P-value                                        0.000       0.000

Full model versus baseline model:

  Comparative Fit Index (CFI)                    0.660       0.677
  Tucker-Lewis Index (TLI)                       0.624       0.642

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)             -18173.075  -18173.075
  Loglikelihood unrestricted model (H1)     -16935.410  -16935.410

  Number of free parameters                        180         180
  Akaike (AIC)                               36706.150   36706.150
  Bayesian (BIC)                             37340.731   37340.731
  Sample-size adjusted Bayesian (BIC)        36770.110   36770.110

Root Mean Square Error of Approximation:

  RMSEA                                          0.087       0.074
  90 Percent Confidence Interval          0.083  0.091       0.070  0.078
  P-value RMSEA <= 0.05                          0.000       0.000

Standardized Root Mean Square Residual:

  SRMR                                           0.146       0.146

Edit:

The real question is probably not only whether to continue with estimates based of a bad fit, but rather whether the scale is new or not.

Edit2:

Barrett (2007) recommends to not rely on "approximate fit indices" at all.

The criterion used for "fit" is actually an abstract concept in the
majority of SEM models. It is clearly not predictive accuracy. In fact
whether models "approximately fit" with an RMSEA of 0.05 or 0.07 is
a literally meaningless scientific statement.

Bibliography

Barrett, P. Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 2007, 42, 815-824 [PDF]

Best Answer

Some immediate comments:

1) I would not predict from this model, since it has bad fit (CFI should be >.95 and RMSEA <.05).

2) It appears you are exploring and confirming on the same data set. This is capitalizing on chance and generally not recommended. It is surprising that your CFA still has bad fit. If you explore on the same data, CFA fit often is much better.

3) You could try exploratory FA or stepwise CFA with modification indices (Lagrange multipliers) to find a better fitting model. I would do this instead of relying on a bad index like $\alpha$ (it is not a good estimator of scale reliability, in fact it is just one estimator, which may be too optimistic about reliability).

4) In any way, I would recommend cross-validating your model. This means splitting your data set in two, explore on one (EFA, modification idices), and confirm (CFA) on the other. This way you avoid capitalizing on chance in scale exploration.

Related Question