Although I have encountered this issue while programming, I want to ask here, because it is a mere math problem.
The task
I have an arbitrary object placed in a three-dimensional space at a certain location with a certain scale and rotation. I know its initial rotation and orientation. Now, I want to know the new orientation of the object after I rotated it in any dimension.
My approach
My idea is this: After I rotate my object, I know the new rotation for all three axes. I calculate the difference between the current rotation and the initial rotation to know how much the object has rotated. Then, I calculate a rotation matrix with these values and transform my initial orientation vector with it. This way, I should get the new orientation of the object. At least, that's what I thought. Spoiler: Does not work.
Example
When I place my object initially, I know all its properties, including its rotation and orientation. The vectors below are the initial values in my test case. The initial_orientation vector means, the object faces to the right. The values for initial_rotation are the rotations around the X-, Y- and Z-axes.
initial_orientation = 1.00000, 0.00000, 0.00000
initial_rotation = -1.39626, -4.71239, 0.00000
After rotating the object around its local x axis, the new rotation value is:
new_rotation = -1.94626, -4.71239, 0.00000
That means the difference between the new and initial orientation is:
'initial_rotation' - 'new_rotation' = 'rotation_difference'
initial_rotation = -1.39626, -4.71239, 0.00000
new_rotation = -1.94314, -4.71239, 0.00000
rotation_difference = -0.54687, 0.00000, 0.00000
I calculate the rotation_matrix by calling a function from the coding framework. The result is listed below.
rotation_matrix = SharpDX.Matrix.RotationYawPitchRoll(rotation_difference.Y, rotation_difference.X, rotation_difference.Z);
rotation_matrix = 1.00000, 0.00000, 0.00000, 0.00000
0.00000, 0.85415, -0.52002, 0.00000
0.00000, 0.52002, 0.85415, 0.00000
0.00000, 0.00000, 0.00000, 1.00000
Now, I calculate the new vector by transforming the initial orientation with the new rotation matrix:
vector = SharpDX.Vector3.TransformCoordinate(initial_orientation, rotation_matrix);
vector = 1.00000, 0.00000, 0.00000
So, the result equals the input. However, from the object's point of view it now looks a bit upwards.
Question
Where is the mistake in my thoughts?
Best Answer
I don't think it is correct to substract the Euler angles of the rotations in order to get the delta (difference) rotation between them, the reason is that Euler angles aren't in the tangent space of the rotation group $SO(3)$.
Instead you can obtain the delta rotation matrix $\delta R$ multiplying the start rotation matrix $R_s$ with thr final rotation matrix $R_f$ in the following way:
$$\delta R = R_f R_s^T$$
Also, in the example you show that the initial orientation vector happens to be an eigenvector of the rotation matrix. If you really want to capture all possible changes in orientation you need to define not a single orientation vector but a "Frame" i.e., three linearly independent vectors. Usually a Frame is composed by three orthogonal unit vectors (a.k.a, orthonormal vectors). The orientation of those vectors defines completely the orientation of any object.