There is more than one definition of a group, which I guess is where the confusion in the comments lie. (A similar confusion came up here before, in the comments to this question.) The first definition I learned as an undergrad, which I guess is the one the OP is working with, is roughly as follows (this is an amended version of the Wikipedia definition):
A group is a set, $G$, together with an operation $\circ$ which may be applied to pairs of elements in $G$. To qualify as a group, the set and operation, $(G, \circ)$, must satisfy four requirements known as the group axioms:
- Closure. For all $a, b\in G$ their product $a \circ b$ is an element of $G$.
- Associativity. For all $a, b, c \in G$ we have $(a \circ b) \circ c = a \circ (b \circ c)$.
- Identity element. There exists an element $e \in G$ such that, for every element $a \in G$, the equation $e \circ a = a \circ e = a$ holds.
- Inverse element. For each $a \in G$, there exists an element $b \in G$ such that $a \circ b = b \circ a = e$, where $e$ is an (the) identity element.
As you point out, $G$ contains two (equivalence classes) of integers such that their product is not in $G$. For example, if $m=6$ then $G=\{1,2,3,4,5\}$ but $2\times3=6\not\in G$. Hence, $G$ is not a group as it fails the "closure" axiom.
(In conclusion: your proof is fine.)
You also ask:
Can a left coset equal right coset if group is not abelian?
Yes. A subgroup $H$ is normal precisely when its left and right cosets are equal. Non-abelian groups can have normal subgroups. For example, $S_3$ is non-abelian, and if $H=\{1, (123)(132)\}$ then the left cosets are $H$ and $(12)H$, while the right cosets are $H$ and $H(12)$.
In the above example, $H$ is abelian but $G=S_3$ is not. There are other examples where both $H$ and $G$ are non-abelian. For example, if you have come across the alternating group $A_n$, then $A_n$ is a normal subgroup of $S_n$ (so left cosets=right cosets), and $A_n$ is non-abelian when $n>3$. (In the above example we actually have $H=A_3$, and this is cyclic so abelian.)
So why isn't that transformation included in it's symmetry group?
Who says it isn't? In the quoted wiki article we read:
Such a transformation is an invertible mapping of the ambient space which takes the object to itself, and which preserves all the relevant structure of the object.
Most generally the symmetry group of an object is just a set of all auto isomorphisms of that object. The word "relevant" is crucial here and in different context it means different things.
If you look at the square as a set then isomorphism = bijection and your example is valid. If you look at it as a topological space then isomorphism = homeomorphism and your example fails cause switching 2 points is not continuous. If you look at it as a subset of some vector space then isomorphism may mean linear isomorphism. If it is metric space then it may mean isometry and so on and so on...
Best Answer
I think the "blocking" isn't and shouldn't be "[non-zero integers] [modulo $p$]" but rather "[non-zero][integers modulo $p$]"
You have three [integers modulo $3$]. They are $[53], [-216]$ and $[3{,}691]$. $[53]$ and $[3{,}691]$ are non-zero [integers modulo $3$]. And $[-216]$ is not a non-zero [integer modulo $3$].