MATLAB: Sign difference between coeff=pca(X) and [~,~,v] = svd(X)

dimensionality reductionMATLABprincipal component analysissingular value decompositionStatistics and Machine Learning Toolbox

I am experiencing sign differences between my computations for principal component analysis using the pca function and the svd function. Consider this matrix:
X = magic(4)
X =
    16     2     3    13
     5    11    10     8
     9     7     6    12
     4    14    15     1
coeff = pca(X) %this will automatically save only the first few principal components
coeff =
     0.5000    0.6708    0.4458
    -0.5000   -0.2236    0.3573
    -0.5000    0.2236    0.6229
     0.5000   -0.6708    0.5344
Now when I do svd, I get the following results.
x0 = bsxfun(@minus,X,mean(X,1)); %first center the data
[~,~,v] = svd(x0)
v =
   -0.5000    0.6708   -0.4458   -0.3182
    0.5000   -0.2236   -0.3573   -0.7565
    0.5000    0.2236   -0.6229    0.5586
   -0.5000   -0.6708   -0.5344    0.1202
So I get essentially negatives of the first and third principal component. Does anyone know why this is?

Best Answer

What we're seeing here is a sign indeterminancy problem that is often intrinsic to methods such as singular value decomposition or eigendecomposition and that can in some cases impact downstream statistical analysis and interpretation. Briefly, the signs of a given pair of corresponding singular vectors are mathematically indeterminant, as both they and their reflections can satisfy the factorization A = UΣV*. The sign is thus essentially arbitrarily assigned during the implementation of algorithms that carry out SVD, with different SVD algorithms capable of producing different results from the same inputs. In some applications, this doesn't matter, but in other cases, it can pose nontrivial problems (for more on this, see this paper).
Within the pca.m function, MATLAB thus enforces a convention such that the largest component (by absolute magnitude) in each right singular vector/eigenvector will be positive (there are other possible conventions as well that also resolve this ambiguity). The following code accomplishes this in lines 429 to 437 of the pca.m function in version R2019b. However, the built-in svd function does not seem to make any such correction (consider this excerpt from the svd documentation: Different machines and releases of MATLAB can produce different singular vectors that are still numerically accurate. Corresponding columns in U and V can flip their signs, since this does not affect the value of the expression A = U*S*V'.), hence the difference you observed. At any rate, if you apply the following convention to the matrix V returned by svd, you should observe results identical to the coeff matrix returned by pca, inclusive of sign. Also note that since principal components (in keeping with the convention of both Hotelling's original 1933 paper and the nearly universal convention in the statistical literature, I use the term "principal components" to refer to the derived empirical orthogonal variables (whose elements are often called scores) defined wrt the eigenvectors, not to the eigenvectors themselves) can be calculated from the singular vectors, either as A*V or as U*S, you should make sure to apply the convention before calculating the principal components, or apply it directly to the principal components afterwards (as is done below).
% Enforce a sign convention on the coefficients -- the largest element in
% each column will have a positive sign.
[~,maxind] = max(abs(coeff), [], 1);
[d1, d2] = size(coeff);
colsign = sign(coeff(maxind + (0:d1:(d2-1)*d1)));
coeff = bsxfun(@times, coeff, colsign);
if nargout > 1
    score = bsxfun(@times, score, colsign); % scores = score
end
Related Question