The first answer to any question on how to find Euler angles is: don't; use quaternions instead.
If external factors force you to use Euler angles, you could proceed like this: If $X$ and $Y$ are identical, the rotation can be a rotation around $X$ through any angle. Otherwise, the rotation can be a rotation through any axis $A$ in the plane $A\cdot X=A\cdot Y$. If $X$ and $Y$ are antipodal, the rotation must be through an angle $\pi$. Otherwise, that plane is spanned by the two vectors $X+Y$ and $X\times Y$, so you can parametrize the possible axes as
$$A(\xi)=\cos\xi\frac{X+Y}{|X+Y|}+\sin\xi\frac{X\times Y}{|X\times Y|}$$ with $\xi\in[0,\pi[$. The rotation angle must be the angle between the perpendicular projections of $X$ and $Y$ into the plane perpendicular to the axis, which are
$$
\begin{eqnarray}
X_\perp&=&
X - (A\cdot X)A\\
&=&X-\cos\xi\frac{X^2+X\cdot Y}{|X+Y|}A\\
&=&X-\frac12\cos\xi\frac{(X+Y)^2}{|X+Y|}A\\
&=&X-\frac12\cos\xi|X+Y|A
\end{eqnarray}$$
(where I used $X^2=Y^2$), and likewise for $Y_\perp$, so the angle between them is
$$
\begin{eqnarray}
\phi(\xi)
&=&\arccos\frac{X_\perp\cdot Y_\perp}{|X_\perp||Y_\perp|}\\
&=&\arccos\frac{X_\perp\cdot Y_\perp}{X_\perp\cdot X_\perp}\\
&=&\arccos\frac{X\cdot Y -\frac12\cos\xi|X+Y|A\cdot(X+Y)+\frac14\cos^2\xi |X+Y|^2}{X\cdot X -\frac12\cos\xi|X+Y|A\cdot(X+X)+\frac14\cos^2\xi |X+Y|^2}\\
&=&\arccos\frac{X\cdot Y -\frac14\cos^2\xi (X+Y)^2}{X\cdot X -\frac14\cos^2\xi (X+Y)^2}\\
&=&\arccos\frac{\cos^2\xi (X+Y)^2-4X\cdot Y}{\cos^2\xi (X+Y)^2-4X\cdot X}\\
&=&\arccos\frac{\sin^2\theta-\cos^2\theta\sin^2\xi}{\cos^2\theta\cos^2\xi-1}\;,\\
\end{eqnarray}
$$
where $\theta$ is the angle between $X$ or $Y$ and $X+Y$, that is, half the angle between $X$ and $Y$.
Using $A(\xi)$ and $\phi(\xi)$, you can construct the rotation matrices $R(\xi)$ and extract the Euler angles.
There are a lot of questions like this, all slightly different.
My earlier answer to a different question
is closely related to what you need, but the question is different enough
that I thought it better to write an answer a little more applicable
to your case.
I'll use right-handed $x,y,z$ Cartesian coordinates.
I visualize roll, pitch, and yaw using the motion of someone's right hand,
whose thumb and index finger are kept outstretched at right angles.
We start out with index finger of pointing in the direction
$(0,1,0)^T$ (positive $y$ axis)
and thumb pointing in the direction $(0,0,1)^T$ (positive $z$ axis).
We turn the entire hand by the angle $\theta$ (the "yaw" angle)
counterclockwise around the $z$ axis.
We then rotate the hand in the plane now occupied by the index finger
and thumb, turning it by the angle $\phi$ (the "pitch" angle)
with the index finger moving in the direction of the thumb.
At this point the angles $\phi$ and $\theta$ describe the
direction of the index finger in something like geographic coordinates,
where $\phi$ corresponds to latitude and $\theta$ corresponds to longitude.
(Note that these are not the kind of "mathematical" spherical coordinates
where $\phi$ is the angle of a vector relative to the $z$-axis.)
Finally, we rotate the hand using the direction of the index finger as
the axis of rotation, turning the thumb toward the palm of the hand
by an angle $\psi$ (the "roll" angle).
The effect of these motions on a vector "attached" to the hand
can be represented by a $3\times3$ matrix.
We can decompose this matrix into a product of three much simpler matrices.
Each of these three matrices will be a rotation around one of the
principal axes ($x$, $y$, or $z$ axes),
unlike the pitch and roll motions described above,
which were performed around axes relative to an already-rotated hand.
To reproduce the result of the roll, pitch, and yaw on the orientation
of the hand, using rotations only around the principal axes, we
have to do the rotations in reverse order. This is so that we can still
do pitch and roll around the correct axes relative to the hand,
but do them while those axes are still aligned with the principal axes.
We do the "roll" through angle $\psi$ first, using the rotation matrix
$$
R_y(\psi) =
\begin{pmatrix} \cos\psi & 0 & -\sin\psi \\
0 & 1 & 0 \\
\sin\psi & 0 & \cos\psi \end{pmatrix}
$$
(a rotation around the $y$-axis),
then the "pitch" through angle $\phi$, using the matrix
$$
R_x(\phi) =
\begin{pmatrix} 1 & 0 & 0 \\
0 & \cos\phi & -\sin\phi \\
0 & \sin\phi & \cos\phi \end{pmatrix}
$$
(a rotation around the $x$-axis),
and finally the "yaw" through angle $\theta$, using the matrix
$$
R_z(\theta) =
\begin{pmatrix} \cos\theta & \sin\theta & 0\\
-\sin\theta & \cos\theta & 0 \\
0 & 0 & 1 \end{pmatrix}
$$
(a rotation around the $z$-axis).
These are much like the axis-rotation matrices used in several other places
(including my earlier answer), but with a sequence of axes and signs
of the matrix entries suitable to your system.
The best way to understand how this works may be to try several examples,
using simple pitch, roll, and yaw angles such as $\pi/2$ or $\pi/4$,
and confirm that this sequence of rotations around fixed
principal coordinate axes has the same result as the desired sequence of
rotations around axes defined by the orientation of the hand.
This sequence of rotations is equivalent to the single rotation
performed by the matrix product $R_z(\theta)R_y(\phi)R_x(\psi)$.
For example, here's what this rotation does to the direction of
the index finger, $(0,1,0)^T$:
\begin{align}
R_z(\theta) R_x(\phi) R_y(\psi) \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}
&= R_z(\theta) R_x(\phi)
\begin{pmatrix} \cos\psi & 0 & -\sin\psi \\
0 & 1 & 0 \\
\sin\psi & 0 & \cos\psi \end{pmatrix}
\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} \\
&= R_z(\theta) R_x(\phi) \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} \\
&= R_z(\theta)
\begin{pmatrix} 1 & 0 & 0 \\
0 & \cos\phi & -\sin\phi \\
0 & \sin\phi & \cos\phi \end{pmatrix}
\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} \\
&= R_z(\theta) \begin{pmatrix} 0 \\ \cos\phi \\ \sin\phi \end{pmatrix} \\
&= \begin{pmatrix} \cos\theta & \sin\theta & 0\\
-\sin\theta & \cos\theta & 0 \\
0 & 0 & 1 \end{pmatrix}
\begin{pmatrix} 0 \\ \cos\phi \\ \sin\phi \end{pmatrix} \\
&= \begin{pmatrix} \cos\phi \sin\theta \\
\cos\phi \cos\theta \\
\sin\phi \end{pmatrix}, \\
\end{align}
which agrees with your result.
What this same rotation does to the direction of
the thumb, $(0,0,1)^T$, is
\begin{align}
R_z(\theta) R_x(\phi) R_y(\psi) \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}
&= R_z(\theta) R_x(\phi)
\begin{pmatrix} \cos\psi & 0 & -\sin\psi \\
0 & 1 & 0 \\
\sin\psi & 0 & \cos\psi \end{pmatrix}
\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} \\
&= R_z(\theta) R_x(\phi)
\begin{pmatrix} -\sin\psi \\ 0 \\ \cos\psi \end{pmatrix} \\
&= R_z(\theta)
\begin{pmatrix} 1 & 0 & 0 \\
0 & \cos\phi & -\sin\phi \\
0 & \sin\phi & \cos\phi \end{pmatrix}
\begin{pmatrix} -\sin\psi \\ 0 \\ \cos\psi \end{pmatrix} \\
&= R_z(\theta) \begin{pmatrix} -\sin\psi \\
-\cos\psi \sin\phi \\
\cos\psi \cos\phi \end{pmatrix} \\
&= \begin{pmatrix} \cos\theta & \sin\theta & 0\\
-\sin\theta & \cos\theta & 0 \\
0 & 0 & 1 \end{pmatrix}
\begin{pmatrix} -\sin\psi \\
-\cos\psi \sin\phi \\
\cos\psi \cos\phi \end{pmatrix} \\
&= \begin{pmatrix} -\sin\psi \cos\theta - \cos\psi \sin\phi \sin\theta \\
\sin\psi \sin\theta - \cos\psi \sin\phi \cos\theta \\
\cos\psi \cos\phi \end{pmatrix} \\
\end{align}
if I haven't dropped a sign or made some other arithmetic error.
Best Answer
You can reason as follows: multiplying on the left some vector by a transformation matrix has the effect of rotating and translating the coordinate frame to which the point is "attached". If you multiply the result by another matrix on the left, you rotated and translate both the transformed frame and the attached point. The global effect is indeed the chaining of the two transformations, performed in the order $H$ then $R$.