This paper describes exactly what you need.
Read this, and then this.
Then if you read a little deeper, you'll find this:
Here is some EulerAngles Code, which you can have a look at...
this google search also gives lots of other solutions to your problem.
You can actually use these methods to recover the regular rotation angles too, as constructing the rotation matrix is trivial, and you only need to reverse the order of the euler rotations to get the the regular rotation angles.
(where by regular and euler, i just mean intrinsic and extrinsic)
EDIT: just realised i answered my own question by accident...
Matrix rows or columns are traditionally listed under $(x,y,z)$ order.
Cyclically change the pairs under consideration i.e $(x,y)\to(y,z)\to(z,x)$. The pairs $(x,y)$ and $(y,z)$ show up in the same order in the matrix but the $(z,x)$ shows up in reverse in the matrix. That is the cause of apparent discrepancy but really there is no discrepancy.
For example write
$x'=x\cos \alpha - y \sin \alpha$
$y'=x\sin \alpha + y \cos \alpha$
now change $(x,y)\to(y,z)\to(z,x)$ and $\alpha\to \beta \to \gamma$ and write the three matrices to see how $(z,x)$ part gets flipped.
Edit:
If you want them to look alike then give up the matrix notation and instead write
$y'=y\cos \beta - z \sin \beta$
$z'=y\sin \beta + z \cos \beta$
And
$z'=z\cos \gamma - x \sin \gamma$
$x'=z\sin \gamma + x \cos \gamma$
In each instance if you try to write $\left[ \matrix{ x' \cr y' \cr z'}\right]$ in terms of $\left[ \matrix{ x \cr y \cr z}\right]$ you will see that the mystery goes away.
Best Answer
The first answer to any question on how to find Euler angles is: don't; use quaternions instead.
If external factors force you to use Euler angles, you could proceed like this: If $X$ and $Y$ are identical, the rotation can be a rotation around $X$ through any angle. Otherwise, the rotation can be a rotation through any axis $A$ in the plane $A\cdot X=A\cdot Y$. If $X$ and $Y$ are antipodal, the rotation must be through an angle $\pi$. Otherwise, that plane is spanned by the two vectors $X+Y$ and $X\times Y$, so you can parametrize the possible axes as
$$A(\xi)=\cos\xi\frac{X+Y}{|X+Y|}+\sin\xi\frac{X\times Y}{|X\times Y|}$$ with $\xi\in[0,\pi[$. The rotation angle must be the angle between the perpendicular projections of $X$ and $Y$ into the plane perpendicular to the axis, which are
$$ \begin{eqnarray} X_\perp&=& X - (A\cdot X)A\\ &=&X-\cos\xi\frac{X^2+X\cdot Y}{|X+Y|}A\\ &=&X-\frac12\cos\xi\frac{(X+Y)^2}{|X+Y|}A\\ &=&X-\frac12\cos\xi|X+Y|A \end{eqnarray}$$
(where I used $X^2=Y^2$), and likewise for $Y_\perp$, so the angle between them is
$$ \begin{eqnarray} \phi(\xi) &=&\arccos\frac{X_\perp\cdot Y_\perp}{|X_\perp||Y_\perp|}\\ &=&\arccos\frac{X_\perp\cdot Y_\perp}{X_\perp\cdot X_\perp}\\ &=&\arccos\frac{X\cdot Y -\frac12\cos\xi|X+Y|A\cdot(X+Y)+\frac14\cos^2\xi |X+Y|^2}{X\cdot X -\frac12\cos\xi|X+Y|A\cdot(X+X)+\frac14\cos^2\xi |X+Y|^2}\\ &=&\arccos\frac{X\cdot Y -\frac14\cos^2\xi (X+Y)^2}{X\cdot X -\frac14\cos^2\xi (X+Y)^2}\\ &=&\arccos\frac{\cos^2\xi (X+Y)^2-4X\cdot Y}{\cos^2\xi (X+Y)^2-4X\cdot X}\\ &=&\arccos\frac{\sin^2\theta-\cos^2\theta\sin^2\xi}{\cos^2\theta\cos^2\xi-1}\;,\\ \end{eqnarray} $$
where $\theta$ is the angle between $X$ or $Y$ and $X+Y$, that is, half the angle between $X$ and $Y$.
Using $A(\xi)$ and $\phi(\xi)$, you can construct the rotation matrices $R(\xi)$ and extract the Euler angles.