There are two vectors : $A = (\hat i + j + k)$ and $B = (\hat i – \hat j – \hat k)$, where $\hat i$, $\hat j$, and $\hat k$ are unit vectors along $x$, $y$, and $z$ axis respectively. We have to find the angle between these two vectors. Of course the best way to do that is by using ‘scalar product’. Scalar product of these two vectors gives $(-1)$, which is equal to $3\cos\theta$
\begin{align}
\implies && -1 & = 3\cos \theta \\
\implies && \theta & = \arccos (-1/3) = 109° \quad \text{(approx)}
\end{align}
Now if I use vector product, I get $A \times B = (2\hat j – 2\hat k)$, so $|A \times B| = \sqrt{8}$, which is equal to $3\sin\theta$.
\begin{align}
\implies && \sqrt{8} & = 3\sin \theta \\
\implies && \theta & = \arcsin (\sqrt{8}/3) = 70.5° \quad \text{(approx)}
\end{align}
Why aren't these two angles equal? Are they not supposed to be equal?
Best Answer
Your cross-product argument is faulty, because the inverse sine cannot distinguish between angles in the interval $[0,90°]$ and angles in the interval $[90°,180°]$.
The correct angle is that obtained from the scalar-product argument, $\arccos(-1/3) \approx 109°$, and you should be able to verify (numerically, at least) that this angle satisfies $$ \sin\mathopen{}\left(\arccos(-1/3)\right)\mathclose{} = \frac{\sqrt{8}}{3} = \frac{||A\times B||}{||A|| \, ||B||}. $$
The arc-sine, on the other hand, is always restricted to producing angles in the interval $[-90°,90°]$, which means that it reflects that $109.5°$ about the $90°$ mark to produce the $70.5°$ that you observe.
Because of this limitation, your vector-product method is unreliable and it shouldn't be used to calculate angles between vectors.