Hint
1) What is $\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} y & y \\ y & y \end{bmatrix}$?
2) The multiplication of matrices is associative.
3) When you are looking for the identity you want
$$\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} e & e \\ e & e \end{bmatrix}=\begin{bmatrix} x & x \\ x & x \end{bmatrix}$$
Now, do the multiplication on the left, what do you get?
4) With the $e$ from $3)$ solve
$$\begin{bmatrix} x & x \\ x & x \end{bmatrix}\begin{bmatrix} y & y \\ y & y \end{bmatrix}=\begin{bmatrix} e & e \\ e & e \end{bmatrix}$$
for $y$. Again, all you need to do is doing the multiplication...
P.S. In order for this to be a group, you need $x \neq 0$.
P.P.S Since $\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}=2\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$, you can prove that
$$F: \mathbb R \backslash\{0 \} \to G$$
$$F(x) =\frac{x}{2} \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$
is a bijection and it preserves multiplications. Since $\mathbb R \backslash\{0 \}$ is a group it follows that G must also be a group and $F$ is an isomorphism... But this is probably beyond what you covered so far...
With (e) you must be careful: the matrices
$$
\begin{pmatrix}
a & 0 \\ 0 & 0
\end{pmatrix}
$$
are not invertible, hence are not elements of $GL_2(\mathbb R)$.
However, the set of matrices of this type
$$
\left\{
\begin{pmatrix}
a & 0 \\ 0 & 0
\end{pmatrix}
: \ a \in \mathbb R, \ a\ne 0
\right\}
$$
is an Abelian group under matrix multiplication. Its identity is not the usual identity $I_2$...
Best Answer
The determinant of each matrix in $H$ is zero. Thus $H$ is not a subset of $G$. Therefore, it cannot be a subgroup of $G$.
Alternatively, $I\notin H$. Thus it cannot be the subgroup of $G$.
Even if $H$ were in $G$, we have the following theorem.
Proof: Let $s\in S$. Then $s\in G$ and
$$\begin{align} sf=s&\implies (s^{-1}s)f=s^{-1}s\\ &\implies ef=e\\ &\implies f=e. \square \end{align}$$