Your question leads to the following fact: If $L \subset \mathbb R^2$ is a lattice spanned by integer vectors, then the number of integer points in $\mathbb R^2 / L$ is the covolume of $L$ (area of a fundamental parallellogram for $L$).
This may be deduced either from
- Pick's formula: Call $i$ the number of interior lattice points of $P$ and $b$ the number of points on the boundary. Call $x$ the number of points on one side and $y$ the number on an adjacent side. Then $b = 2x+2y-4$ (we counted the four corners twice). Then the number of lattice points in $\mathbb R^2 / L$ is $i + x + y - 3$: By taking $x+y$, we counted the point in the corner four times, whereas we only want it once. By Pick's formula, the volume of $P$ is $i + b/2 - 1$. The two are equal: $i+b/2 - 1 = i + (2x+2y-4)/2-1 = i + x + y - 3$.
- The following intuitive argument (which can be made hard). Let $P$ be a fundamental parallellogram for $L$ and $n$ the number of integer points in $\mathbb R^2 / L$. Tile together $N^2$ copies of $P$ to obtain the parallellogram $NP$ for some large $N \in \mathbb N$. The volume of $NP$ is $N^2 \cdot \mathrm{Vol}(P)$, so the number of integer points in this region is $N^2 \cdot \mathrm{Vol}(P) + O(N)$. On the other hand, the number of integer points is $N^2 \cdot n + O(N)$. Taking $N\to \infty$, we see that we must have $n = \mathrm{Vol}(P)$.
- The following argument: Let $B$ be a matrix that sends $\mathbb Z^2$ to $L$. Then $B$ gives a linear map $\mathbb Z^2 \to \mathbb Z^2$ and we want to count the index $[\mathbb Z^2 : B(\mathbb Z^2) ]$. From the Smith normal form for $B$, we see that this index equals $|\det(B)|$.
So you don't need the matrix $A$ to be hyperbolic, you also don't need $|\det A| = 1$; all you need is that $A - I$ is invertible, which is the case if $A$ is hyperbolic.
The solution is based on the Fourier series representation in $L^2$ space. In fact, the following important lemma of mixing helps to solve the problem a lot.
Lemma: $T$ is mixing in measure space $(X, \mathcal{L}, \mu)$ iff for any $L^2$ basis of $X$ ${\{\xi_i\}}$ we have
$$\lim_N\int_X \xi_i\circ{T^N}\xi_jd\mu = \int_X \xi_i d\mu \int_X \xi_j d\mu$$
Then apply this lemma to the Fourier basis $\{\xi_{m,n}=e^{mx+ny}\}_{m,n \in \mathbb{Z}}$ we WTS
$$\lim_N\int_X \xi_{m,n}\circ{T^N}\xi_{k,l}d\mu = \int_X \xi_{m,n} d\mu \int_X \xi_{k,l} d\mu$$
and discuss by two classifications $(m,n) = 0$ and $(m,n) \neq (0,0)$ to get the solution. Note that $\int_X \xi_{m,n}d\mu = 0$ iff $(m,n) \neq 0$ and $\int_X \xi_{m,n}d\mu=1$ iff $(m,n) = 0$. In the case $(m,n) \neq 0$ we can find enough large $N$ such that for any $n > N$ we have $\xi_{m,n}\circ T^n \neq \xi_{k,l}$ for any $(k,l)$, hence the LHS vanishes.
The idea is not belong to me but I think it is easy to learn.
Best Answer
Here is a low-brow argument:
Consider the eigenspaces $S$ and $U$ of the matrix $A$, corresponding to the eigenvalues with modulus less than $1$ and greater than $1$, respectively. Take small segments $S_\epsilon$ and $U_\epsilon$ (of positive length) of the eigenspaces around the origin; for $\epsilon$ small enough the product $\mathcal{S}_{0,\epsilon}\times \mathcal{U}_{0,\epsilon}$ of their images under the quotient map $\mathbb{R}^2\to\mathbb{T}^2$ will be a neighborhood of $0$ in the torus. The set $\mathcal{S}_{0,\epsilon}$ is mapped (strictly) into itself under $A$, and any element in $\mathcal{S}_{0,\epsilon}\setminus\{0\}$ converges to $0$ exponentially fast, hence can not be recurrent.