$\newcommand{\R}{\mathbb{R}}$To pick up from the comment of @NaN, look for an element
$$
E = \begin{bmatrix}
e & e\\
e & e\\
\end{bmatrix}
$$
such that for each $a \ne 0$ we have
$$
\begin{bmatrix}
a & a\\
a & a\\
\end{bmatrix} \cdot E
=
\begin{bmatrix}
a & a\\
a & a\\
\end{bmatrix}
$$
And then you'll find that the inverse is not quite the one you wrote...
Hint 3 below explains the logic behind this exercise.
Hint 1
$e = 1/2$
Hint 2
The inverse of $$\begin{bmatrix}a & a\\a & a\\\end{bmatrix}$$ is $$\begin{bmatrix} a^{-1}/4 & a^{-1}/4\\ a^{-1}/4 & a^{-1}/4\\\end{bmatrix}$$.
Hint 3
The logic behind this is that the given matrices form a semigroup which is isomorphic to the non-zero reals under the operation $a * b = 2 a b$. So you may forget about matrices and think about the latter. This, in turn, is obtained by transport of structure from the usual multiplication, under the map $f(x) = 2x$, that is, $f : (\R^{\star}, *) \to (\R^{\star}, \cdot)$ is an isomorphism.
First of all you have to ask yourself if it is possibile that $H/K$ is a group, i.e if $K$ is normal in $H$.
In your case (please verify this fact) you have that $K$ is an abelian group (in particular is isomorphic to $(\mathbb{R},+,0)$ ) and is contained in the center of $H$ (I.e $AK=KA$ for each $A \in H$, and $K\in K$), so that $K$ is clearly normal in $H$.
Thus The quotient set $H/K=\{AK: A \in H\}$ inherits also a group structure.
Now we have to understand what is the multiplication in $H$ to compute easily the quotient set $H/K$.
Taking an element $A\in H$, I will denote $A$ as $A(a,b,c)$, where $A_{12}=a, A_{13}=b$ and $A_{23}=c$.
Then taking two matrices $A=A(a,b,c)$ and $B=B(a’,b’,c’)$, you have
$AB=AB(a+a’, b+ac’ +b’, c+c’)=BA$
and so if you have a general element $A=A(a,b,c)$, you can observe that $A’(a,0,c) $ is an element in its class on $H/K$, in fact
$A(a,b,c)=A’(a, 0, c)K(0, b+ac’+b’, 0) $
So that you can define the section map $s: H/K \to G$ Sending each class $AK$ to $A’(a,0,c)$, where $(G:=\{ A(a,0,c) : a,c \in \mathbb{R}\},+,0)$ is an additive group.
You can prove that this map is an isomorphism of groups and observing that $A(a,0,c)+B(a’,0,c’)=(A+B)(a+a’, 0, c+c’)$
then you can say that $H/K \cong (\mathbb{R}^2,+,0)$
Best Answer
Well you can start by claiming the two products are equal and then this will determine your coefficients. For example you can immediately see that $b$ must be equal to $c$, and you should be able to get the other relations from here.