Hint
1) What is $\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} y & y \\ y & y \end{bmatrix}$?
2) The multiplication of matrices is associative.
3) When you are looking for the identity you want
$$\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} e & e \\ e & e \end{bmatrix}=\begin{bmatrix} x & x \\ x & x \end{bmatrix}$$
Now, do the multiplication on the left, what do you get?
4) With the $e$ from $3)$ solve
$$\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} y & y \\ y & y \end{bmatrix}=\begin{bmatrix} e & e \\ e & e \end{bmatrix}$$
for $y$. Again, all you need to do is doing the multiplication...
P.S. In order for this to be a group, you need $x \neq 0$.
P.P.S Since $\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}=2\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$, you can prove that
$$F: \mathbb R \backslash\{0 \} \to G$$
$$F(x) =\frac{x}{2} \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$
is a bijection and it preserves multiplications. Since $\mathbb R \backslash\{0 \}$ is a group it follows that G must also be a group and $F$ is an isomorphism... But this is probably beyond what you covered so far...
Best Answer
It is a monoid (and hence also a semigroup).
First, we have to show that $I_2 \in G$. Indeed, setting $a=1$ and $b=0$ gives the identity matrix.
Second, we have to show that $G$ is closed under multiplication. Suppose that $\begin{bmatrix} a&-b \\ b&a \end{bmatrix} \in G$ and $\begin{bmatrix} c&-d \\ d&c \end{bmatrix} \in G$. Then, we have
$$\begin{bmatrix} a&-b \\ b&a \end{bmatrix}\begin{bmatrix} c&-d \\ d&c \end{bmatrix}=\begin{bmatrix} ac-bd&-(ad+bc) \\ ad+bc&ac-bd \end{bmatrix} \in G$$, so $G$ is closed under multiplication.
Finally, multiplication of matrices is associative in general, so it is in particular associative for matrices in $G$.
Alternatively, we could instead use the fact that $G$ is isomorphic to the monoid of complex numbers under multiplication via the map $\begin{bmatrix} a&-b \\ b&a \end{bmatrix} \mapsto a+bi$, and the fact that multiplication of complex numbers is associative.