Hint
1) What is $\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} y & y \\ y & y \end{bmatrix}$?
2) The multiplication of matrices is associative.
3) When you are looking for the identity you want
$$\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} e & e \\ e & e \end{bmatrix}=\begin{bmatrix} x & x \\ x & x \end{bmatrix}$$
Now, do the multiplication on the left, what do you get?
4) With the $e$ from $3)$ solve
$$\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} y & y \\ y & y \end{bmatrix}=\begin{bmatrix} e & e \\ e & e \end{bmatrix}$$
for $y$. Again, all you need to do is doing the multiplication...
P.S. In order for this to be a group, you need $x \neq 0$.
P.P.S Since $\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}=2\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$, you can prove that
$$F: \mathbb R \backslash\{0 \} \to G$$
$$F(x) =\frac{x}{2} \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$
is a bijection and it preserves multiplications. Since $\mathbb R \backslash\{0 \}$ is a group it follows that G must also be a group and $F$ is an isomorphism... But this is probably beyond what you covered so far...
Best Answer
To check if $S:=\left\{\begin{bmatrix}\cos(x)&-\sin(x)\\\sin(x)&\cos(x)\end{bmatrix}:x\in\Bbb R\right\}$ is closed under multiplication you would need to multiply $$\begin{bmatrix}\cos(x)&-\sin(x)\\\sin(x)&\cos(x)\end{bmatrix}\begin{bmatrix}\cos(y)&-\sin(y)\\\sin(y)&\cos(y)\end{bmatrix}$$ not $$\begin{bmatrix}\cos(x)&-\sin(x)\\\sin(x)&\cos(x)\end{bmatrix}\begin{bmatrix}\cos(x)&-\sin(x)\\\sin(x)&\cos(x)\end{bmatrix}$$ because the two matrices need not be the same. So we need to show $$\begin{bmatrix}\cos(x)&-\sin(x)\\\sin(x)&\cos(x)\end{bmatrix}\begin{bmatrix}\cos(y)&-\sin(y)\\\sin(y)&\cos(y)\end{bmatrix}=\begin{bmatrix}\cos(x)\cos(y)-\sin(x)\sin(y)&-\cos(x)\sin(y)-\sin(x)\cos(y)\\\sin(x)\cos(y)+\cos(x)\sin(y)&-\sin(x)\sin(y)+\cos(x)\cos(y)\end{bmatrix}$$ is in $S$. Recall the identities: $$(1)\qquad\sin(x\pm y)=\sin(x)\cos(y)\pm\sin(y)\cos(x)\\(2)\qquad \cos(x\pm y)=\cos(x)\cos(y)\mp \sin(x)\sin(y)$$