all you need to do is measure how the vectors vec3(1,0,0) vec3 (0,1,0) vec3(0,0,1) get changed by any matrix.
vec3(1,0,0)'s transformation is equal to the first column of the matrix
vec3(0,1,0)'s transformation is equal to the second column of the matrix
vec3(0,0,1)'s transformation is equal to the second column of the matrix (only if it has 3 dimensions)
vec4(0,0,0,1)'s transformation is equal to the second column of the matrix (only if it has 4 dimensions),...
yes, learn how matrices function on basic core level:
in a 2d to 2d matrix transformation, 2 matrix colums are equal to what the 2 base-vectors vec2(1,0) and vec2(0,1) are changed into by the matrix. any scalar of that is also transformed linearily scaled to that, as a matrix transformation is a linear transformation.
in a 3d to 3d matrix transformation, same goes for 3 base-vectors vec3(1,0,0) vec3 (0,1,0) vec3(0,0,1) being changed by 3 matrix columns.
in a 3d to 2d matrix, 3 columns of 2 lines tell how 3 3d basis vectors as in the above are scamed in 3d space, as a sum of three 2d vectors (that get scaled by the matrix).
Correct, that matrix is representing the quaternion rotation.
Yes, you can perform rotations with quaternions without converting quaternions to matrices. Since you can embed your Euclidean 3D vectors into quaternion 4D linear space, just considering the $q_w=0$, you can rotate your vectors using quaternion product:
$v' = q v q^*$
Where $q^*$ is the conjugate quaternion. Then you can map back the rotated 4D vector $v'$ to the 3D space.
Best Answer
Suppose that your transformation consists of a rotation by non-zero angle $\theta$ (about the origin), whose $3 \times 3$ matrix we denote $R$, followed by the translation $x \mapsto x + d$. That is, the full transformation (in standard coordinates) is given by $T(x) = Rx + d$.
Let $J$ denote the $90^\circ$ rotation about the same axis as $R$. Note that $R$ can be written in the form $R = \cos \theta I + \sin \theta J$, where $I$ denotes the $3 \times 3$ identity matrix. Its "square root" can be written in the form $S = \cos (\theta/2)I + \sin (\theta/2)J$. It is easy to see that $S$ should be the rotational component of our "halfway" transformation.
Note that if we are given the transformation matrix $R$, then the angle satisfies $$ \operatorname{trace}(R) = 1 + 2 \cos \theta, $$ which allows us to compute $\cos \theta, \sin \theta,$ and $J$ from the matrix $R$.
We will describe the halfway transformation as $H(x) = Sx + e$, and we wish to solve for the vector $e$. We must select $e$ such that $T(x) = H(H(x))$. That is, $$ Rx + d = S(Sx + e) + e = S^2x + Se + e = Rx + (I + S)e. $$ In other words, we must select $e$ such that $(I+S)e = d$. That is, it suffices to take $e = (I + S)^{-1}d$, where we note that $(I + S)$ must be invertible because it is a rotation by $\theta/2$, which is not a multiple of $180^\circ$ (so there is no $x$ for which $Sx = -x \implies (I + S)x = 0$).
We can go a step further an compute this inverse explicitly. We have $$ (I + S)^{-1} = ((1 + \cos (\theta/2))I + \sin(\theta/2)J)^{-1}. $$ It turns out that because we have $J^2 = -I$, we can treat these coefficients as the real and complex parts of a complex number. So, following the process by which one would compute $(a + bi)^{-1}$, we find that $$ (I + S)^{-1} = \frac{1}{(1 + \cos(\theta/2))^2 + \sin^2(\theta/2)}((1 + \cos (\theta/2))I - \sin(\theta/2)J) \\ = \frac{1}{2 + 2 \cos (\theta/2)}((1 + \cos (\theta/2))I - \sin(\theta/2)J), $$ which is to say that we have $$ e = \frac{1}{2 + 2 \cos (\theta/2)}((1 + \cos (\theta/2))I - \sin(\theta/2)J)d. $$