It means that the unexplained variance from two variables are correlated. One way of thinking of this is as a partial correlation.
Say we have two regression equations:
\begin{equation}
Y_1i=\beta1_1 \ X_i+\epsilon1_i
\end{equation}
\begin{equation}
Y_2i=\beta2_1 \ X_i+\epsilon2_i
\end{equation}
Both equations have an $\epsilon$ term. If you model that as two equations, that's fine. But what if you model it as one equation - do you want to assume that the $\epsilon$ terms are uncorrelated? If you do, then don't correlate them - as in, don't put estimate a correlation in the residual. Usually you don't, so you'd correlate the residuals.
An example: Say you want to look at the effect of age (in adults) on: speed at running 100m, speed at running 5 miles. I'd expect a negative relationship for both of these, but if you modeled them in one equation, you'd expect unexplained variance in 100m running speed to be correlated with 5 mile running speed, controlling for age - so the residuals are correlated.
You can also think of this in terms of latent variables - there are common causes of the residual for both 100m and 5 mile speeds, and hence you can hypothesize the existence of a latent (unmeasured) variable.
Main Point
With only two observed variables per factor, the latent variable can not generally be estimated. Having correlations between factors presumably adds enough constraints to the model to allow estimation of the latent factors, but without those correlations, the latent factors are not estimatable.
What should you do?
- Add more observed variables per factor. If you have three or more observed variables, your latent factors will generally be estimatable.
- Constrain factor loadings to be equal. As a secondary option, you could constrain you factor loadings for the two items to be equal. Make sure they are on the same scale so that this makes sense.
I think the option of having at least three observed variables per factor is preferable.
Note also that setting the covariance between each factor to 1 will not create uncorrelated factors. If the factors have unit variance (i.e., like z-scores), then your factors will perfectly correlated (r=1). Thus, it would be equivalent to having a single factor. If the factors do not have unit variance, then it be constraining some other specific correlation structure that is related to the variance of each of the factors.
Comments below relate to the original post prior to the update.
Degrees of freedom
- If you constrain a parameter to be a constant, then you should get an additional degree of freedom in an SEM. Thus, when you specify $k$ latent factors to be uncorrelated, then that implies you are specifying $k(k-1)/2$ potential parameters to be zero and you should have that many additional degrees of freedom.
- If you are constraining $k$ latent factors to have a common covariance, then you should get $k(k-1)/2 - 1$ additional degrees of freedom, because you are still estimating one parameter.
So why might your degrees of freedom not be changing? Here are a few thoughts:
- Have you included the second order factor in your model testing the first-order model? If you have, remove the second-order factor.
- Have you specified constraints correctly in Amos?
Constraining factor correlations to be equal
- Note that in social sciences, it is often the correlation between factors that is conceptualised to be approximately the same from a theoretical perspective. Thus, you may want to ensure that your factors are all on the same scale before constraining covariances. For example, you could constrain the covariance of the factors to be one, rather than constraining one of the loadings, as is the default in Amos. By placing the factors on the same scale, constraining covariances to be equal is the same as constraining correlations to be equal.
Negative variances
- In terms of errors, it may be that the factors are so highly correlated that forcing the factors to be uncorrelated is causing estimation problems.
- You may want to just try removing the double headed arrows in Amos rather than specifying constraints to see whether that makes a difference to see whether that makes a difference
- You may have issues with number of items per factor.
There's also some discussion here
Best Answer
The questions you ask are those of identifying the scale of the latent variables. One possible set of answers is:
F1is given byQ; the scale ofF2is given byC; etc. Adding a fixed variance to that will definitely lead to weird results.Another possible set of answers is:
That way, your latent variable is scaled by its residual term, and you don't have to agonize over the best measure to use as the scaling factor.
See also Gonzalez and Griffin 2001, although I think they simply stumbled upon the finite sample sensitivity of Wald tests to reparameterization... a not-terribly-well-known phenomenon even among well-trained statisticians.
Negative values of error variances are spooky indeed. These are sometimes called Heywood cases. Ken Bollen and I have summarized what we knew about them in this paper.