I think your problem is related to frame of reference. Your global frame is A and other frames obtained after rotation in A are local frames (such as B). So u want all your rotations in global frame but they are taking place in local frame. For rotation in global frame of reference you need to pre-multiply the transformation matrix and for rotation in local frame, post-multiply. For eg, if u want rotation by $\theta_1$ in A for which transformation matrix is $R_{\theta_1}$ and then rotation by $\theta_2$ in A (rotation matrix $R_{\theta_2}$), then:
$$V_f = R_{\theta_2}*R_{\theta_1}*V_i$$
where $V_i$ is initial position of the point(or object) and $V_f$ is final.
for further reference, check this link:
Maths - frame-of-reference for combining rotations
The most straightforward way of calculating rotations in 3D is by using Euler angles and matrix multiplication. I hope you are familiar with matrix multiplication.
Let's start by taking a vector $\vec{v}$ in 3 dimensions. This can be any vector of your choise. In the end, we will take $\vec{v} = z$-axis. Now, if we take any matrix $R$ whose determinant is $\det{R}=1$ and calculate
$$
R\vec{v}
$$
it turns out that this is some rotation of the original vector. It points to a new direction, and the length does not change.
The simplest rotation is a rotation about the $x$-axis, with an angle $\alpha$. Omitting the proof, it can be shown that the matrix
$$
R_x(\alpha) = \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & \cos{\alpha} & -\sin{\alpha} \\
0 & \sin{\alpha} & \cos{\alpha} \\
\end{array}
\right]
$$
can be used for this. Maybe you have seen something like this before? I suggest that you carry out some example calculations with some vectors of your choise. Do you see how the $x$-coordinate of the rotated vector doesn't change when you multiply them? Okay, good.
The other elementary rotation matrices are of course around the $y$ and $z$-axes. As a summary, all of the three elementary matrices are
\begin{equation}
R_X(x) = \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & \cos{x} & -\sin{x} \\
0 & \sin{x} & \cos{x} \\
\end{array} \right]
\qquad
R_Y(x) = \left[
\begin{array}{ccc}
\cos{x} & 0 & \sin{x} \\
0 & 1 & 0 \\
-\sin{x} & 0 & \cos{x} \\
\end{array} \right]
\qquad
R_Z(x) = \left[
\begin{array}{ccc}
\cos{x} & -\sin{x} & 0 \\
\sin{x} & \cos{x} & 0 \\
0 & 0 & 1 \\
\end{array} \right]
\end{equation}
These rotation matrices can then be combined. The most common Euler angle rotation order is the extrinsic $xyz$-order, which produces the (a bit complex-looking)
\begin{equation}
\begin{split}
R &= R_Z(\psi) R_Y(\theta) R_X(\phi) \\
&=
\left[
\begin{array}{ccc}
\cos{\psi} & -\sin{\psi} & 0 \\
\sin{\psi} & \cos{\psi} & 0 \\
0 & 0 & 1 \\
\end{array} \right]
\left[
\begin{array}{ccc}
\cos{\theta} & 0 & \sin{\theta} \\
0 & 1 & 0 \\
-\sin{\theta} & 0 & \cos{\theta} \\
\end{array} \right]
\left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & \cos{\phi} & -\sin{\phi} \\
0 & \sin{\phi} & \cos{\phi} \\
\end{array} \right]
\\
&=
\left[
\begin{array}{ccc}
\cos{\theta}\cos{\psi} & \cos{\psi} \sin{\theta} \sin{\phi} - \cos{\phi} \sin{\psi} & \cos{\phi} \cos{\psi} \sin{\theta} + \sin{\phi} \sin{\psi} \\
%
\cos{\theta}\sin{\psi} & \cos{\phi} \cos{\psi} + \sin{\theta} \sin{\phi} \sin{\psi} & \cos{\phi}\sin{\theta} \sin{\psi} - \cos{\psi} \sin{\phi} \\
%
-\sin{\theta} & \cos{\theta}\sin{\phi} & \cos{\theta}\cos{\phi}
\end{array}
\right]
\end{split}
\end{equation}
This just means that the first rotation is carried out about the original $x$ axis (rotation by angle $\phi$), the second rotation is carried out about the original $y$-axis (angle $\theta$) and the last rotation is carried out about the original $z$-axis (angle $\psi$). The word "extrinsic" means that the axes by which the second and third rotation are carried out, do not change as a function of the previous rotations. So they are the "original" axes that remain fixed during the whole process.
The rotation order that you are requesting is an intrinsic $yxz$-rotation. Note that the previous calculations were only applied to the extrinsic rotation orders, which is the opposite of intrinsic. However, it can be shown that in order to calculate the intrinsic rotation, you only need to reverse the rotation order and then calculate the equivalent extrensic rotation. Proof for this is again omitted. So now we want to have the extrinsic $zxy$ rotation. This is merely a reordering of the previous example, resulting in
\begin{equation}
\begin{split}
R &= R_Y(\theta) R_X(\phi) R_Z(\psi) \\
&=
\left[
\begin{array}{ccc}
\cos{\theta} & 0 & \sin{\theta} \\
0 & 1 & 0 \\
-\sin{\theta} & 0 & \cos{\theta} \\
\end{array} \right]
\left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & \cos{\phi} & -\sin{\phi} \\
0 & \sin{\phi} & \cos{\phi} \\
\end{array} \right]
\left[
\begin{array}{ccc}
\cos{\psi} & -\sin{\psi} & 0 \\
\sin{\psi} & \cos{\psi} & 0 \\
0 & 0 & 1 \\
\end{array} \right]
\\
&=
\left[
\begin{array}{ccc}
\cos{\theta}\cos{\psi} + \sin{\theta}\sin{\phi}\sin{\psi} & - \cos{\theta}\sin{\psi}+\cos{\psi}\sin{\theta}\sin{\psi} & \cos{\phi}\sin{\theta} \\
\cos{\phi}\sin{\psi} & \cos{\phi}\cos{\psi} & - \sin{\phi}\\
-\cos{\psi}\sin{\theta} + \cos{\theta}\sin{\phi}\sin{\psi} & \sin{\theta}\sin{\psi} + \cos{\theta}\cos{\psi}\sin{\phi} & \cos{\theta}\cos{\phi}
\end{array}
\right]
\end{split}
\end{equation}
(I recommend you double-check this calculation, because it's very easy to make a mistake in the multiplication). In your case, $\psi=0$, simplifying the result a bit:
$$
R =
\left[
\begin{array}{ccc}
\cos{\theta} & \sin{\theta}\sin{\phi} & \cos{\phi}\sin{\theta} \\
0 & \cos{\phi} & - \sin{\phi} \\
-\sin{\theta} & \cos{\theta}\sin{\phi} & \cos{\theta}\cos{\phi}
\end{array}
\right]
$$
You were asking about the new components of the $z$-axis, in particular, so we must multiply this with the corresponding vector:
$$
R \vec{z} =
\left[
\begin{array}{ccc}
\cos{\theta} & \sin{\theta}\sin{\phi} & \cos{\phi}\sin{\theta} \\
0 & \cos{\phi} & - \sin{\phi} \\
-\sin{\theta} & \cos{\theta}\sin{\phi} & \cos{\theta}\cos{\phi}
\end{array}
\right]
\left(
\begin{array}{c}
0 \\ 0 \\ 1
\end{array}
\right)
= \left(
\begin{array}{c}
-\sin{\theta} \\ \cos{\theta}\sin{\phi} \\ \cos{\theta}\cos{\phi}
\end{array}
\right)
$$
Huh, quite a calculation! So just plug in $\theta=a$ and $\phi=b$ to get the result.
Best Answer
You want to rotate the surface so that the direction $(1,0,0)$ becomes $\dfrac1{\sqrt3}(1,1,1)$. A possible rotation matrix is
$$\begin{pmatrix}\frac1{\sqrt3}&0&-\frac2{\sqrt6} \\\frac1{\sqrt3}&\frac1{\sqrt2}&\frac1{\sqrt6} \\\frac1{\sqrt3}&-\frac1{\sqrt2}&\frac1{\sqrt6}\end{pmatrix}$$
Now your equation is
$$\frac{(u+v+w)^2}3-\frac{(v-w)^2}2-\frac{(-2u+v+w)^2}6=1$$
or
$$u^2+v^2+w^2-4uv-4vw-4wu+3=0.$$
Caution: sign and permutations errors are guaranteed, I didn't check. But the final result must not be far from correct.
https://www.wolframalpha.com/input/?i=plot+x%5E2%2By%5E2%2Bz%5E2-4xy-4yz-4zx%2B3%3D0