linear algebralinear-transformationsmatricesorthogonal matricesreflection
Are these two definitions of a reflection matrix Q equivalent (if and only if)?
Definition 1: $Q^TQ = I$ and $det(Q) = -1$
Definition 2: $Q = I-2nn^T$ where $n$ is a unit normal vector to the plane that reflects vectors.
It’s easy to show that definition 2 implies 1 (I’ve already done this – so please do not prove this part). But proving that definition 1 implies 2 seems much more difficult.
It looks like you’ve either misunderstood or misquoted this part of the paper. The product of two rotations is another rotation, but the corresponding axis has a rather complicated relationship to the two component rotation axes. A simple counterexample: rotate 90° about the $x$-axis and then 90° about the $z$-axis. The result is equivalent to a 120-degree rotation about $(1,1,1)$, which is certainly not parallel to $(0,0,1)\times(1,0,0)$.
The identity in the paper is actually $A_jA_{j'}^T\mathbf m=\mathbf m$, where the $A$’s are reflections rather than rotations: $A=(I-2\mathbf n\mathbf n^T)R$. The rotations $R$ are the same for both $A$’s and they cancel, leaving a product of two pure reflections.
What this identity says is that the composition of two reflections is a rotation about an axis that’s perpendicular to both of the normals of the reflecting planes. This is a standard result. In fact, the rotation axis is the intersection of these planes: each reflection fixes points that lie on its reflecting plane, so the fixed points of the composition are those points that lie on both planes. (If the planes are parallel, the composition is instead a translation, which might be viewed as a rotation about a line at infinity.) It turns out that the rotation angle is twice the dihedral angle of the planes, but that doesn’t appear to be used in the paper. The parenthetical remark in the paper about this product being a “special orthogonal matrix with 2 complex conjugate eigenvalues and 1 eigenvalue equal to 1” is just a long-winded way of saying that it is a rotation.
Partial Answer -- Existence:
I remember running into this problem a while ago as part of an (old) qualifier.
My solution was to use Householder reflectors. Let $H$ be a hyperplane in $\mathbb{R}^n$, and $v$ a normal vector to $H$, such as in the diagram below. Then the matrix
$$F = I - 2 \frac{vv^\ast}{v^\ast v}$$
reflects $\mathbb{R}^n$ across $H$. Visually,
(Figure from Trefethen & Bau's Numerical Linear Algebra.)
There are some details about how one might choose $v$ and its sign, and how to apply this more broadly (to e.g. QR factorization), but those details are beyond the scope of this post.
So, if you wish to reflect $x$ to $y$ and vice versa, you need to find the corresponding Householder reflector that sends $x$ to $y$. Householder reflectors are projectors (i.e. $F^2=I$), so it is enough to find such an $F$ where $Fx=y$.
The natural starting point is the midpoint. If $x:=(x_i)_{i=1}^n, y := (y_i)_{i=1}^n$, then consider $$ m := \left( \frac{x_i+y_i}{2} \right)_{i=1}^n $$ $m$ is the midpoint, and should be fixed by the transformation $F$ (i.e. $Fm=m$). One can easily rationalize that, from the geometry, that $m$ lies in the hyperplane of concern, and between $x$ and $y$ (which are not in the hyperplane).
One can then see what the natural choice of $v$ is: we should have a vector $v$ where $$ m+v=x $$ so $$ v = x-m $$ giving us a Householder reflector by the noted formula.
However, I myself am unsure on uniqueness and how to best verify that.
Best Answer
The result is wrong for real vector spaces with dimension $n>2$
The orthogonal transformation
$$Q= \begin{pmatrix} \dfrac{\sqrt{2}}{2} & -\dfrac{\sqrt{2}}{2} & 0\\ \dfrac{\sqrt{2}}{2} & \dfrac{\sqrt{2}}{2} & 0\\ 0 & 0 &-1 \end{pmatrix}$$
is such that $\det Q= - 1$. However, $Q$ is the not the matrix of a reflection ($Q$ has no fixed vector).