I am working with the spin 1 representation of SU(2), which is just SO(3). The ordinary generators used in quantum mechanics are:
$J_x = \left(
\begin{array}{ccc}
0 & \frac{1}{\sqrt{2}} & 0 \\
\frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\
0 & \frac{1}{\sqrt{2}} & 0 \\
\end{array}
\right)$; $J_y = \left(
\begin{array}{ccc}
0 & -\frac{i}{\sqrt{2}} & 0 \\
\frac{i}{\sqrt{2}} & 0 & -\frac{i}{\sqrt{2}} \\
0 & \frac{i}{\sqrt{2}} & 0 \\
\end{array}
\right)$; $J_z = \left(
\begin{array}{ccc}
1 & 0 & 0 \\
0 & 0 & 0 \\
0 & 0 & -1 \\
\end{array}
\right)$.
But I can also work with just the ordinary 3D rotation matrices that rotate vectors specified by (x,y,z) components. From taking the infinitesimal limit of these matrices, I find that the generators are:
$J_x' = \left(
\begin{array}{ccc}
0 & 0 & 0 \\
0 & 0 & -i \\
0 & i & 0 \\
\end{array}
\right); J_y' = \left(
\begin{array}{ccc}
0 & 0 & i \\
0 & 0 & 0 \\
-i & 0 & 0 \\
\end{array}
\right); J_z' = \left(
\begin{array}{ccc}
0 & -i & 0 \\
i & 0 & 0 \\
0 & 0 & 0 \\
\end{array}
\right)$.
After exponentiating, I get rotation operators $D(\psi,\theta,\phi)$ and $D'(\psi,\theta,\phi)$, where the $D$ operators act on vectors written in the j=1, m =-1,0,1 basis and the $D'$ operators act on vectors written in a different basis.
What is the relation between these two operators (or between the generators)? Are they related by a unitary transformation? If so, how does one go about finding that unitary transformation? (That is what I first suspected, but I was unable to find any unitary transformation that works.) Or are they not related in a direct way?
EDIT:
Corrected sign error in generators.
Best Answer
As per the suggestion of Phoenix87, we can find the unitary transformation by noticing that the transformation that diagonalizes $J_z'$ will also transform $J_z'$ to into $J_z$ (since $J_z$ is diagonal). So we find the eigenvectors of $J_z'$ and then use them as the columns of our unitary transformation. We are free, however, to choose the phase of the eigenvectors, and so we make the choice that gives $U^\dagger J_x' U = J_x$ and $U^\dagger J_y' U = J_y$. When all is said and done, we obtain:
$U = \left( \begin{array}{ccc} -\frac{i}{\sqrt{2}} & 0 & \frac{i}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\ 0 & i & 0 \\ \end{array} \right)$.
Which gives $U^\dagger J_i' U = J_i$.