You're fourth step is a bit of an overkill, in terms of testing whether a subset is a subgroup. Indeed, we need to check:
- $(1)$ the identity element of $G$, namely $e = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$ is in $H$ and that
- $(2)$ for two elements $h_1, h_2 \in H$, $h_1\cdot h_2 \in H$,
But with respect to testing for inclusion of inverses in H (whether for every element $h \in H$, we also have that $h^{-1} \in H)$, we don't need to test for inverses of products of elements, since you've shown $H$ is closed under multiplication.
Indeed, you are correct that for most $h$ with integer entries $a, b$, the inverse $h^{-1} \notin H$ because the corresponding non-zero entries will likely be rational numbers, but not both integers.
So, taking for example $h = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$, when we compute $h^{-1} = \begin{bmatrix} \frac 12 & 0 \\ 0 & \frac 12\end{bmatrix}$, we've shown that $h^{-1} \notin H$, because $\frac 12 \notin \mathbb Z$.
Hence, we've shown that $H$ is not closed under taking inverses. (All you need to provide is one counterexample of some element $h \in H$ such that $h^{-1} \notin H$, to show that $H$ is not closed under taking inverses.) And we therefore conclude that $H$, as defined, is NOT a subgroup of $G$. And then you're done.
With (e) you must be careful: the matrices
$$
\begin{pmatrix}
a & 0 \\ 0 & 0
\end{pmatrix}
$$
are not invertible, hence are not elements of $GL_2(\mathbb R)$.
However, the set of matrices of this type
$$
\left\{
\begin{pmatrix}
a & 0 \\ 0 & 0
\end{pmatrix}
: \ a \in \mathbb R, \ a\ne 0
\right\}
$$
is an Abelian group under matrix multiplication. Its identity is not the usual identity $I_2$...
Best Answer
The identity under multiplication is 1. The recriprical (multiplicative inverse) of $x + yi$ is $\frac {x}{x^2 + y^2} - \frac {y}{x^2 + y^2}i$.