Let $X\in\mathbb{R}^{3\times 3}$ be an orthogonal matrix. Then $\mathrm{vec}X\in\mathbb{R}^9$ is a 9 by 1 vector formed by stacking the columns of the matrix $X$ on top of one another. Given a matrix $A\in\mathbb{R}^{9\times 9}$, find the optimal orthogonal matrix $X$ minimizing the following objective function.
$$J=\left(\mathrm{vec}X\right)^T A \mathrm{vec}X$$
I think Kronecker product may be useful for solving this problem. Does a closed-form solution exist? If not, is it possible to solve it iteratively? Thanks.
EDIT：
a) Here $A$ doesn't have any special property. But it is also acceptable if solutions can be obtained by adding some properties on $A$.
b) In the original problem, $X$ is constrained as a rotation matrix. But I think that would be even harder, so I put $X$ as an orthogonal matrix herein. Of course, optimal rotation matrices are better.
Best Answer
This is a bit late, but I believe this might help.
Use the Cayley parametrization of orthogonal matrices. Example usage in optimization: A Feasible Method for Optimization with Orthogonality Constraints
Broken link: http://www.caam.rice.edu/~wy1/paperfiles/Rice_CAAM_TR10-26_OptManifold.PDF
You might want to start by just understanding the Wiki article on the Cayley transform and why a parametrization helps (you can do the optimisation without constraints, which should be easier)