Suppose that the rotation matrix is defined as $\mathbf{R}$. Then in order to rotate a vector and a matrix, the following expressions are, respectively, used
$\mathbf{u'}=\mathbf{R} \mathbf{u}$
and
$\mathbf{U'}=\mathbf{R} \mathbf{U} \mathbf{R}^T$,
where $\mathbf{u}$ and $\mathbf{U}$ are, respectively, an arbitrary vector and an arbitrary matrix.
For me, the first one is obvious since you simply multiply the rotation matrix by the vector (for example a point coordinate in 3D) and obtain the rotated vector (rotated point coordinate in 3D). However, the second one is not clear for me and why the rotation should be multiplied from both sides and how this expression is derived.
P.S. The matrix $\mathbf{U}$ can be interpreted as a stretch matrix in 3D.
Best Answer
Here the matrix $U$ is considered not as a bunch of column vectors, but as a (matrix of the) linear map $F\colon {\Bbb R}^n\to {\Bbb R}^n$ $$ y=F(x)=Ux. $$ What happens if we rotate both $y$ and $x$ by $R$? We get (since $R^TR=I$ for rotations) $$ y=Ux\quad\Rightarrow\quad Ry=RUx\quad\Rightarrow\quad Ry=\underbrace{RUR^T}_{U'}Rx\quad\Rightarrow\quad y'=U'x'. $$ Thus the matrix $U'=RUR^T$ corresponds to the same linear map $F$ in the new coordinates after rotation ($x'\mapsto y'$).
In general, for any change of the basis $x'=Sx$, $y'=Sy$ the corresponding change of the matrix $U$ is $$ Sy=\underbrace{SUS^{-1}}_{U'}Sx\quad\Rightarrow\quad y'=U'x'. $$ It means that the class of all similar matrices $\{SUS^{-1}\colon S\text{ invertible}\}$ is exactly the class of all matrices that describe the same linear map in different bases.