$<a^3>$ is $\{a^3, a^6, a, a^4, a^7, a^2, a^5, 1\} = C_8$. So clearly, $a^6$ and $a^4$ belong to the same coset. Same with the second problem.
$<a^3b> = \{a^3b, 1\}$ because $a^3ba^3b = a^3a^{-1}ba^2b = a^2ba^2b = abab = b^2 = 1$. The cosets are $\left\{\{a^3b, 1\}, \{a^4b, a\}, \{a^5b, a^2\}, \{a^6b, a^3\}, \{a^7b, a^4\}, \{b, a^5\}, \{ab, a^6\}, \{a^2b, a^7\}\right\}$.
The keep this question from being unanswered...
Let $G$ be a finite group generated by $x$ and $y$, with $x$ and $y$ of order $2$. We want to show that $G\cong D_{2n}$ (the dihedral group of order $2n$), where $n$ is the order of $xy$.
To answer your questions first: in the dihedral group $D_{2n}=\langle r,s\mid r^n = s^2 = 1, sr=r^{-1}s\rangle$, every element not in $\langle r\rangle$ is of order $2$. To verify this, note that every element can be written uniquely as $r^is^j$, with $0\leq i\lt n$, $0\leq j\lt 2$. The elements not in $\langle r\rangle$ are precisely the ones with $j=1$. Such an element satisfies:
$$\begin{align*}
(r^is)^2 &= r^i(sr^i)s\\
&= r^i(r^{-i}s)s &\text{(since }sr=r^{-1}s\text{)}\\
&= r^0s^2\\
&= 1.
\end{align*}$$
Thus, all such elements are elements of order $2$.
When $n$ is odd, these are the only elements of order $2$; when $n$ is even, all of these are elements of order $2$, and so is $r^{n/2}$. So in a dihedral group, you always have at least half the elements of order $2$.
If you think of the dihedral group as the symmetries/rigid motions of a regular $n$-gon sitting on the plane inscribed in the unit circle, you have several axes through which you can reflect the polygon, not just the $x$-axis. The bisection through each vertex gives you a line through which you can reflect the polygon, getting an element of order $2$.
Now, the proof of the desired statement. We note that $xy$ and $y$ satisfy the relations in the presentation of $D_{2n}$: indeed, by definition of $n$ we know that $(xy)^n = 1$; and $y^2=1$ by assumption. Finally, we have that
$$\begin{align*}
y(xy) &= (yx)y\\
&= (y^{-1}x^{-1})y &\text{(since }x^2=y^2=1\text{)}\\
&= (xy)^{-1}y.
\end{align*}$$
By von Dyck's Theorem there is a homomorphism $f\colon D_{2n}\to G$ mapping $r$ to $xy$ and $s$ to $y$. Under this homomorphism, $x$ is the image of $rs$.
(There are other possible homomorphisms, since the map sending $r\mapsto r^i$, $s\mapsto s$, with $\gcd(i,n)=1$, is an automorphism of $D_{2n}$, so pre-composing it with $f$ gives you a slightly different map).
The link you give in your question contains several different answers showing that this $f$ is indeed an isomorphism.
Best Answer
I assume, you have also noted that $i\in\{0,1\}$ when writing an element as $r^it^j$. Essentially, there are just four cases to consider:
The elements of $D_\infty$ are $t^j\colon x\mapsto x+j$ with $j\in \mathbb Z$, i.e. translations by an integer amount, and $rt^j\colon x\mapsto -x-j$, i.e. reflection around integers or half-integers (note that $-\frac j2$ is the fix pint of $x\mapsto -x-j$).
Bonus Question: Can you find any element of $G$ that is not in $D_\infty$?
Remark: How could you show that all elements have the form $r^it^j$ without observing how elemnts of $D_\infty$ are multiplied?