Over a finite field of order $q$, it is easy to count matrices with nonzero determinant (and from there, get matrices with zero determinant by complementary counting).
There are $q^n-1$ choices for the first column: we just have to choose a nonzero vector in $\mathbb F_q^n$. Then there are $q^n-q$ choices for the second column: we must avoid choosing any multiple of the first column. There are $q^n-q^2$ choices for the third column (since there are $q^2$ distinct linear combinations of the first two columns, which we must avoid, and so on). So the number of $n\times n$ matrices over $\mathbb F_q$ with nonzero determinant is
$$
\prod_{k=0}^{n-1} (q^n - q^k)
$$
and then $q^{n^2}$ minus this product gives the number of matrices with zero determinant.
But $\mathbb Z_4$ (if by this you mean the integers modulo $4$) is not a field, and so this calculation doesn't quite work. For example,
$$\det \begin{bmatrix}2 & 0 \\ 0 & 2\end{bmatrix} \equiv 0 \pmod 4$$ even though the second column is not a multiple of the first.
Here is a general formula for $n\times n$ matrices over $\mathbb Z_4$.
Write the matrix as $A + 2B$, where $A,B$ have entries from $\{0,1\}$.
Suppose that mod $2$, $A$ has corank $r$ (rank $n-r$). Then we can choose $r$ independent (mod $2$) vectors $\mathbf x_1, \dots, \mathbf x_r$ such that $A\mathbf x_i \equiv \mathbf 0 \pmod 2$. Extend these to a set of $n$ vectors that are independent mod $2$, and let $X$ be the matrix whose columns are these $n$ vectors.
Because $X$ is invertible mod $2$, $X$ is invertible mod $4$, so $\det(A + 2B) = \det(X(A+2B)X^{-1}) = \det(XAX^{-1} + 2 XBX^{-1})$. If we find a decomposition of $XAX^{-1} + 2 XBX^{-1}$ into $A' + 2B'$, where $A', B'$ have entries from $\{0,1\}$, then the first $r$ rows of $A'$ are $0$ and the last $n-r$ rows are linearly independent.
It follows that if $r \ge 2$, then $\det(A + 2B)$ is definitely $0$: a row expansion by the first row of $A' + 2B'$ picks up a factor of $2$, and a row expansion by the second row of $A' + 2B'$ picks up another factor of $2$. What about when $r=1$?
When $r=1$, we can factor out a $2$ from the first row of $A' + 2B'$. This lets us rewrite $\det(A' + 2B')$ as $2 \cdot \det(A'' + 2B'')$ where:
- the first row of $A''$ is the first row of $B'$, and other rows are taken from $A'$.
- the first row of $B''$ is $0$, and other rows are taken from $B'$.
If $\det(A' + 2B') = 2$, then $\det(A'' + 2B'')$ is odd, which means $A''$ is invertible mod $2$. Given the last $n-1$ rows of $A''$ (which were fixed by $A'$), $2^{n-1}$ out of $2^n$ choices for the first row of $A''$ (which comes from $B'$) will make $A''$ invertible mod $2$. Therefore half of the possible choices for $B'$ (which come from half of the possible choices for $B$) give determinant $2$ here. So exactly half of the matrices with $r=1$ will have determinant $0$, and the other half will have determinant $2$.
Thus, if we want to count the matrices mod $4$ with determinant $0$, let $N_0, N_1, N_{\ge 2}$ (with $N_0 + N_1 + N_{\ge 2} = 2^{n^2}$) be the number of matrices over $\mathbb F_2$ with corank $0$, corank $1$, and corank $\ge 2$. Then the number we're interested in is
$$
N_1 \cdot 2^{n^2-1} + N_{\ge 2} \cdot 2^{n^2}.
$$
(When $A$ is counted by $N_{\ge 2}$, any matrix $B$ will make $\det(A+2B)=0$; when $A$ is counted by $N_1$, half of all possible matrices will do.) For example, when $n=2$, $N_1 = 9$ and $N_{\ge 2} = 1$, so there are $9 \cdot 2^3 + 1 \cdot 2^4 = 72 + 16 = 88$ matrices.
We have $N_0 = \prod_{k=0}^{n-1} (2^n - 2^k)$ by the formula for the first section. Counting $N_1$ is a bit harder, but we can do it: if we first choose a linearly dependent row on the $i^{\text{th}}$ step ($i=0, 1,\dots, n-1$), the number of ways is $2^i \prod_{k=0}^{n-2} (2^n - 2^k)$, so altogether $N_1 = (2^n - 1)\prod_{k=0}^{n-2} (2^n - 2^k)$. So the general formula for $n \times n$ matrices of $\mathbb Z_4$ is
$$
2^{n^2-1}(2^n - 1)\prod_{k=0}^{n-2} (2^n - 2^k) + 2^{n^2}\left(2^{n^2} - \prod_{k=0}^{n-1} (2^n - 2^k) - (2^n - 1)\prod_{k=0}^{n-2} (2^n - 2^k)\right).
$$
Some algebra simplifies this to
$$
4^{n^2} - 2^{n^2-1} (2^{n+1}-1) \prod_{k=0}^{n-2} (2^n - 2^k).
$$
The sequence begins $1, 88, 100864, 1735131136, \dots$.
For $\mathbb Z_{p^2}$ in general, similar reasoning should work.
The given answer is certainly wrong, but you've overlooked some cases too. If one of the rows is $aaa$ and another row is $bbb$ these two rows are linearly dependent and the determinant is $0$.
Also, consider the matrix $$\begin{bmatrix}b&b&b\\a&b&a\\b&a&b\end{bmatrix}$$ The determinant is $0$ since the sum of the last two rows is a constant multiple of the first row.
I get $248$ by running the problem through sympy:
from sympy import symbols, Matrix
from itertools import product
a,b = symbols('a,b')
count = 0
for p in product({a,b}, repeat = 9):
M = Matrix([p[:3],p[3:6],p[6:]])
if M.det().expand() == 0:
print(M)
count += 1
print(count)
The program prints $248.$ (It also prints out all the singular matrices, but I don't suppose anyone wants to see that.)
I agree with your calculation of $176.$ I haven't tried to verify that the two other cases I've identified account for all $72$ missing singular matrices. I leave that for you.
EDIT
I have verified that these two cases account for the other $72$ singular matrices ($36$ in each case) but I'll let you fill in the details. They're easy.
EDIT
To count the case where one row is $aaa$ and another $bbb$ note that the third row must be different from $aaa$ and $bbb$ or we would have counted the matrix already. That gives $6$ choices for the third row, and then there are $3!$ ways to order them, giving $36.$
For the second case, we have one row with two $a$s and a $b$. There are three possible such rows. Once we have chosen this row, the row with two $b$s and one $a$ is determined since the sum of the two rows must be a constant vector. The third row must be $aaa$ or $bbb$ so we have $6$ choices, and $36$ matrices as before.
Best Answer
Matrices with $\det <0$ are exactly those resulting from swapping the first two rows of matrices with $\det >0$ because this multiplies the determinant by $-1$, and so the number with $\det >0$ is the same as the number with $\det <0$. So you just need to count the matrices with $\det =0$.
First we count how many of the $2^4 =16$ matrices of the form $$A = \begin{pmatrix} 1 & 1 & 1 \\ 1 & a & b \\ 1 & c & d \end{pmatrix}$$ have $\det A =0$.
The determinant of $A$ is $$\det A = \left| \begin{smallmatrix} a & b \\ c & d \end{smallmatrix} \right| - \left| \begin{smallmatrix} 1 & b \\ 1 & d \end{smallmatrix} \right| + \left| \begin{smallmatrix} 1 & a \\ 1 & c \end{smallmatrix} \right| = (ad -a -d) - (bc -b -c).$$
We have the first term equal to $3$ when $(a,d)=(-1,-1)$, and equal to $-1$ in the other three cases- and the same for the second term. We therefore have one in the case $a=b=c=d=-1$ and $3 \times 3 = 9$ for the other cases, totaling $10$ such matrices $A$.
Each matrix with determinant $0$ can be uniquely made into the above form by multiplying a choice of the rows and columns by $-1$ (which doesn't change the determinant). There are $10$ matrices of determinant $0$ for each of the $2^5 = 32$ ways to choose those entries in the first row and column.
Hence, out of the $2^9 = 512$ matrices with entries in $\{ 1, -1 \}$, $320$ have $\det = 0$, $96$ have $\det > 0$ and $96$ have $\det <0$.