As you've demonstrated in 1), the question boils down to when $Ax=0$ has a non-trivial solution. It turns out that this is the case if and only if $\det A$ is a zero divisor. I've written this up in a separate post because it's of interest in its own right: necessary and sufficient condition for trivial kernel of a matrix over a commutative ring. It follows that the answer to your question is yes, $A$ is a zero divisor if and only if $\det A$ is a zero divisor, and thus $R^{n\times n}$ inherits from $R$ the property that all non-zero elements are either units or zero divisors.
What should be correct way to solve it?
Proving the given statement (with the finiteness conditon) would be a duplicate question of Every nonzero element in a finite ring is either a unit or a zero divisor, so I will direct you to the very good solutions that already exist there.
What happens if we drop "finite" condition on R?
$\Bbb Z$ is a commutative ring with unity which has exactly two units and (in my convention) only one zero divisor, and everything else is neither unit nor zero divisor. So deleting the word finite pretty much wrecks the statement.
I know I am wrong but I thought $R$ to be an integral domain.
Well my example just now was an integral domain, but no, being a domain has nothing to do with it. The ring $F[x]/(x^2)$ for an infinite field $F$ is an infinite commutative ring with identity which isn't a domain and yet is partitioned between units and zero divisors.
So you can't just delete finite, but you can easily replace it with far more general adjectives.$^\ast$ Being finite is in no way necessary for elements to be partitioned between units and zero divisors. There are huge classes of infinite rings that have that property.
Probably the next simplest generalization of this question is to prove that for any right Artinian ring $R$, every element is a unit or zero divisor (I count $0$ among zero divisors.) Finite rings are of course left and right Artinian. But now you have, for example, every $n\times n$ matrix ring over a field, and every quotient of a polynomial ring over a field as examples beyond finite rings.
Actually you can go to something even more general called a strongly $\pi$ regular ring$^{\ast\ast}$. There is a simple proof for both claims at this question: Rings whose elements are partitioned between units and zero-divisors. and here.$^{\ast\ast\ast}$
$^\ast$ You can simply delete the word commutative, though.
$^{\ast\ast}$ A strongly $\pi$-regular ring is one which has the descending chain condition on chains of the form $xR\supseteq x^2R\supseteq x^3R\supseteq\ldots$
$^{\ast\ast\ast}$ There is also a related (but more oddly worded) post about this.
Best Answer
Lemma. Every surjective endomorphism $f : M \to M$ of a finitely generated $R$-module $M$ is an isomorphism.
Proof: $M$ becomes an $R[x]$-module, where $x$ acts by $f$. By assumption, $M=xM$. Nakayama's Lemma implies that there is some $p \in R[x]$ such that $(1-px)M=0$. This means $\mathrm{id}=p(f) f$. Hence, $f$ is injective. $\square$
Corollary: If $f,g$ are endomorphisms of a finitely generated $R$-module satisfying $fg=\mathrm{id}$, then also $gf=\mathrm{id}$.