I found this exercise in the book "Introduction to Linear Algebra" by Serge Lang. I suppose it's expected to be done using the formula for the angle between two vectors:
$$ \cos \theta = \frac{A \cdot B}{||A|| \ ||B||} $$
This is my process:
First, I name the points:
$$\begin{aligned}
&A = (2,-1,1) \\
&B = (1, -3, -5) \\
&C = (3, -4, -4)
\end{aligned}$$
Now, the sides:
$$\begin{aligned}
&X = \overrightarrow{AB} = B – A = (-1, -2, -6) \\
&Y = \overrightarrow{BC} = C – B = (2, -1, 1) \\
&Z = \overrightarrow{AC} = C – A = (1, -3, -5)
\end{aligned}$$
Now I get the needed values
$$\begin{aligned}
&X \cdot Y = -2 +2 – 6 = -6 \\
&Y \cdot Z = 2 + 3 – 5 = 0 \\
&X \cdot Z = -1 + 6 + 30 = 35
\end{aligned}$$
$$\begin{aligned}
&||X|| = \sqrt{(-1)^2 + (-2)^2 + (-6)^2} = \sqrt{41} \\
&||Y|| = \sqrt{2^2 + (-1)^2 + 1^2} = \sqrt{6} \\
&||Z|| = \sqrt{1^2 +(-3)^2 + (-5)^2} = \sqrt{35}
\end{aligned}$$
Now the cosines
$$\begin{aligned}
&\cos \angle XY = \frac{X \cdot Y}{||X|| \ ||Y||} = -\frac{6}{\sqrt{41}\sqrt{6}} = -\frac{6}{\sqrt{246}} \\
&\cos \angle YZ = \frac{Y \cdot Z}{||Y|| \ ||Z||} = 0 \\
&\cos \angle XZ = \frac{X \cdot Z}{||X|| \ ||Z||} = \frac{35}{\sqrt{41}\sqrt{35}} = \frac{35}{\sqrt{1435}}
\end{aligned}$$
Now the angles to check if it's right
$$\begin{aligned}
&\angle XY \approx 112.49º &(?) \\
&\angle YZ = 90º \\
&\angle XZ \approx 22.5º
\end{aligned}$$
Yet the angles don't add to $180º$ so it can't be a triangle. The problem is with the angle of $XY$, if the product $X \cdot Y$ were positive it would have come all right. However, since it's negative, it's the cosine of an angle on the 2nd quadrant. To my understanding, if the dot product of two vectors is negative then the angle between them is over $90º$, however it's stated in the problem that the shape is a triangle.
Feels like I made some stupid mistake but I've been looking at this exercise for close to an hour without knowing what I did wrong.
Best Answer
You get the angles highlighted in red when you select the vectors as you've done.
Now, $\angle A = $ angle b/w $\vec X$ and $\vec Z \approx 22.49^\circ$. (taken 2 digits)
But,
$\angle B \approx 180 - 112.49 = 67.51^\circ$
$\angle C = 180 - 90 = 90^\circ$
Now,
$22.49$(approx) $+ 67.51$(approx) $ + 90 = 180$
Showing they add to $180^\circ$ without approximation.
$\angle B = 180^\circ - \arccos\left(-\sqrt{\frac{6}{41}}\right) = \arccos \left(\sqrt{\frac{6}{41}}\right) $ (because, $\arccos(x) = 180^\circ - \arccos(-x)$)
$\angle A = \arccos\left(\sqrt{\frac{35}{41}}\right)$
$\begin{align}\angle A +\angle B +\angle C&= \arccos\left(\sqrt{\frac{35}{41}}\right) + \arccos \left(\sqrt{\frac{6}{41}}\right)+90^\circ \\ &=\arccos\left(\sqrt{\frac{6}{41}}\sqrt{\frac{35}{41}} - \sqrt{1-\left(\frac{6}{41}\right)^2}\sqrt{1-\left(\frac{35}{41}\right)^2}\right)+90^\circ\\ & = \arccos\left(\sqrt{\frac{6}{41}}\sqrt{\frac{35}{41}}-\sqrt{\frac{35}{41}}\sqrt{\frac{6}{41}}\right)+90^\circ \\ &=\arccos(0)+90^\circ \\ & = 180^\circ \end{align}$