Martin Brandenburg asked in a comment how one might come up with the proof in the paper linked by YACP and transcribed in Jonathan's answer; I'll try to answer that here by showing the train of thought that led me to the same (as far as I can see) proof. When I thought about it, I had see the "edit" of the question, using $1+b$ in place of $b$ in the unital case, and, as you'll see, that contributed to my approach.
First, notice that the given equation $(ab)^2=a^2b^2$ asserts a sort of commutativity, since the left side $abab$ and the right side $aabb$ differ only in the order of a pair of $a,b$ factors. The equation can be rewritten to emphasize this observation: $a(ba-ab)b=0$ or, reversing the sign $a(ab-ba)b=0$. So the commutator $ab-ba$ is killed when sandwiched between $a$ and $b$. Also when sandwiched between $b$ and $a$ (see the equation before I reversed the sign, and switch the roles of $a$ and $b$).
The next idea is to try to push this idea of killer sandwiches further. If I could express the same commutator $ab-ba$ as the commutator of some other two elements, then those elements would also kill it in a sandwich. Here the $1+b$ idea surfaced. The commutator of $a$ and $1+b$ is the same as that of $a$ and $b$, because $1$ commutes with $a$. Unfortunately, we don't have a unity element $1$ in our ring. But we do have some other things that commute with $a$ and therefore might be used instead of $1$ here. The simplest of these is $a$ itself. So the commutator of $a$ and $b$ is also the commutator of $a$ and $a+b$ and is therefore killed when sandwiched by $a$ and $a+b$.
But if you write down the two equations saying that $ab-ba$ is killed when sandwiched by $a,b$ and also when sandwiched by $a$ and $a+b$, and if you subtract the first of these equations from the second, you get that $ab-ba$ is killed when sandwiched by $a$ and $a$. Symmetrically or similarly, it's also killed when sandwiched by $b$ and $b$.
These results combine to say that $ab-ba$ is killed when sandwiched by any two (non-commutative) polynomials in $a$ and $b$ as long as these have no constant terms. In particular, it is killed when sandwiched by itself and itself. That means its cube is zero. But we're given that there are no non-zero nilpotents in our ring, so the commutator itself has to be zero.
I hope that, at least by using more words and fewer complicated equations, I've given an idea how a more-or-less reasonable human being might arrive at this proof.
Best Answer
I like your idea of trying matrices over $\mathbb{Z}/2\mathbb{Z}$. However, you don't want to take all matrices, as you have two many idempotents, $x$ such that $x^2 = x$.
For instance $a = E_{11} = \left[\begin{smallmatrix}1&0\\0&0\end{smallmatrix}\right]$ and $b=E_{11}+E_{12} = \left[\begin{smallmatrix}1&1\\0&0\end{smallmatrix}\right]$. In general $E_{ij} \cdot E_{jk} = E_{ik}$ and if $j\neq k$ then $E_{ij} \cdot E_{k\ell} = 0$. Hence $ab=b$ but $ba=a$; of course $a^2 = a$ and $b^2=b$.
So the hint is to take a subring of the matrix ring that is not commutative but that avoids having too many idempotents. Idempotents are like partial 1s and partial 0s. What we want is for our elements to either be “1” or “0” not both. In more standard language, to avoid idempotents we make sure every element is either a unit ($x$ such that $x^{-1}$ exists) or nilpotent ($x$ such that $x^n=0$).
The very first idea along these lines doesn't quite work, but is a good idea: we take a basis with one element a unit (the identity matrix $E_{11}+E_{22}$) and the other element nilpotent ($x=E_{12}$). It is not too hard to show that this vector space is closed under multiplication, so we do get a ring. And in this ring $(ab)^2 = (ba)^2$ for all $a,b$. However, this ring consists of only $0,1,x,1+x$ and it is not hard to check it is commutative; it is $\mathbb{Z}[x]/(2,x^2)$.
You can fix the commutativity problem without too much trouble though: