This is just the cosine rule:
$\cos \theta = {\|u\|^2 +\|v\|^2 - \|u-v\|^2 \over 2 \|u\| \|v\| } = {52 +{5 \over 4} - {149 \over 4} \over 2 \sqrt{ 52 } \sqrt{{5 \over 4} } }$.
You are missing a square root or two.
Here's one way to look at it. Start with the circle $S^1=\lbrace (x,y):x^2+y^2=1\rbrace$. Apply the function $f$ to $S^1$. The circle is all points $(x,y)$ who satisfy the circle equation, right? So therefore $f(S^1)$ is the set of points $f(x,y)$, where $(x,y)$ satisfies the circle equation.
$$\text{Image}=f(S^1)=\lbrace f(x,y): x^2+y^2=1\rbrace=\lbrace (x-2y,x-3y): x^2+y^2=1\rbrace$$
So this says that if you find a pair of real numbers $u$ and $v$ which can be written in this funky way $(u,v)=(x-2y,x-3y)$ AND if $(x,y)$ satisfies the circle equation, then it is in the image. That's kind of bizarre. You'd rather know what the condition on $u$ and $v$ are directly. Well, given $u=x-2y$ and $v=x-3y$ it is not that hard to rewrite the equation $x^2+y^2=1$. In fact, you get $x=3u-2v$ and $y=u-v$, which is what you have.
Let's look at this a different way now, because the above method is kind of the naive thing to do. The one encouraged by your book is not so bad. Lets think of the image as follows: every point in $f(S^1)$ is the image of a point on the circle. Thus, if $(x,y)$ is in the image, then this means that $(u,v)=f^{-1}(x,y)$ is a pair of points satisfying the circle equation.
$$u^2+v^2=1$$
But, we know what $f^{-1}$ is. It's just a function like $f$. You compute that $f^{-1}(x,y)=(3x-2y,x-y)$. Thus $(u,v)=(3x-2y,x-y)$ substituting this gives the result.
Best Answer
Yes, this matrix has a special property, namely its determinant is 1.