I'm trying to find the cosets in
$G/K$ and write down the multiplication table of $G/K$ for
$G = ⟨a⟩ × ⟨b⟩$, where $o(a) = 8$ and $o(b) = 2$, and $K = \langle (a^2, b) \rangle$.
(In other words, $G=C_8×C_2$, the cartesian product of the cyclic groups of order 8 and 2, respectively)
I am familiar with the idea of cosets $Ha = \{ha \mid h \in H\}$ for right cosets generated by $a$ and $aH = \{ah \mid h \in H\}$ for left cosets generated by $a$.
Also to clarify, I am not referencing the $a$ used in the definition of group $G$ here, just giving an example of left and right coset definition.
I am not familiar with the process of finding cosets for quotient groups.
Best Answer
EDIT. Initially I interpreted the subgroup $K$ as being generated by two elements $a^2$ and $b$, answering accordingly. I have changed the answer to address the question where $K$ is generated by a single element $a^2b$.
First, the group $G$. If you think of $G$ as the product $C_8 \times C_2$, then the elements are of the form $(a,b)$, where \begin{equation} \def\cor#1{\color{red}{#1}} \def\cog#1{\color{darkgreen}{#1}} \def\cob#1{\color{darkblue}{#1}} \begin{aligned} \cor{a^8} &\cor{{}= e} \qquad&& \text{($a$ has order $8$)}, \\ \cog{b^2} &\cog{{}= e} \qquad&& \text{($b$ has order $2$)}, \\ \cob{ab} &\cob{{}= ba} \qquad&& \text{($a$ and $b$ commute)}. \end{aligned} \label{rels} \tag{ * } \end{equation}
But you can also just use $a$ and $b$ as generators and multiply them in words like $a$, $ab$, $a^5$, $babbabab$, whatever.
These are equivalent by the correspondence (isomorphism of groups) \begin{array}{c@{50em}cc} G &\longleftrightarrow& C_8 \times C_2 \\ e && (e,e) \\ a && (a,e) \\ b && (e,b) \\ a^ib^j && (a^i,b^j) \end{array}
In $G$, the elements \begin{array}{*8{l}} e, &a, &a^2, &a^3, &a^4, &a^5, &a^6, &a^7, \\ b, &ab, &a^2b, &a^3b, &a^4b, &a^5b, &a^6b, &a^7b \end{array} are each distinct, and any word in $a$ and $b$ is equivalent to one of these by the relations (\ref{rels})
For example, \begin{align} \def\cor#1{\color{red}{#1}} \def\cog#1{\color{darkgreen}{#1}} \def\cob#1{\color{darkblue}{#1}} ba^{-6}b^{-1}ab^3 &= bea^{-6}b^{-1}ab^3 = b\cor{a^8}a^{-6}b^{-1}ab^3 = ba^2b^{-1}ab^2b \\ &= ba^2b^{-1}a\cog{e}b = ba^2b^{-1}ab = \cob{a^2b}b^{-1}ab = a^2eab = a^2ab = a^3b \end{align} which is one of those $8 \cdot 2 = 16$ elements of $G$
Notice how the exponents in $a^ib^j$ are just the totals for each of the exponents in the original words: $$ i = (-6)+1 = -5 \equiv 3 \pmod{8} \qquad\text{and}\qquad j = 1+(-1)+3 = 3 \equiv 1 \pmod{2} $$
Now the subgroup $K = \langle a^2b \rangle < G$. Let's work out the left cosets of $K$. (But since this group is abelian, these are also the right cosets.)
Since $K$ is generated by $a^2b$, elements of the subgroup consist of powers of $(a^2b)^i = a^{2i}b^{i}$, where the exponent of $a$ is considered mod $8$ and the exponent of $b$ is considered mod $2$, i.e. \begin{array}{*8{l}} \def\coy#1{\color{lightgray}{#1}} e, &\coy{a,} &\coy{a^2,} &\coy{a^3,} &a^4, &\coy{a^5,} &\coy{a^6,} &\coy{a^7,} \\ \coy{b,} &\coy{ab,} &a^2b, &\coy{a^3b,} &\coy{a^4b,} &\coy{a^5b,} &a^6b, &\coy{a^7b} \end{array} The subgroup $K$ is the identity coset, but any of these elements can be a representative of the coset (it goes by many names): $$ K = eK = a^2bK = a^4K = a^6bK. $$ Now, consider the coset $aK$, consisting of elements $a^{2i+1}b^i$: \begin{array}{*8{l}} \def\coy#1{\color{lightgray}{#1}} \coy{e,} &a, &\coy{a^2,} &\coy{a^3,} &\coy{a^4,} &a^5, &\coy{a^6,} &\coy{a^7,} \\ \coy{b,} &\coy{ab,} &\coy{a^2b,} &a^3b, &\coy{a^4b,} &\coy{a^5b,} &\coy{a^6b,}, &a^7b \end{array} This coset also has $4$ names, using any of the elements as a representative: $$ aK = a^3bK = a^5K = a^7bK. $$
There are $2$ more cosets: $$ a^2K = a^4bK = a^6K = bK, $$ and $$ a^3K = a^5bK = a^7K = abK. $$
That's it! Since $\lvert G \rvert = 16$ and $\lvert K \rvert = 4$, we only expect $16/4 = 4$ cosets. In other words, the subgroup has index $4$, denoted $[G: K] = 4$.
This means the quotient group $G/K$ has order $4$. Explicitly, the elements of $G/K$ are $\bigl\{ K, aK, a^2K, a^3K \bigr\}$, and the multiplication table looks like: \begin{array}{r|r*3{@{2em}r}} & K & aK & a^2K & a^3K \\ \hline K & K & aK & a^2K & a^3K \\ aK & aK & a^2K & a^3K & K \\ a^2K & a^2K & a^3K & K & aK \\ a^3K & a^3K & K & aK & a^2K \end{array}
You don't have to take my word for it. For example, since a typical element in $K$ looks like $a^{2i} \, b^i$, where $i \in \mathbb{Z}$, a typical element in $a^mK$ looks like $a^{2i+m} \, b^i$, so we can compute, e.g., $a^mK \, a^nK$ by multiplying generic elements in the cosets: $$ \bigl( a^{2i+m} b^i \bigr) \, \bigl( a^{2j+n} b^j \bigr) = a^{2(i+j)+m+n} \, b^{i+j} = a^{m+n} \, a^{2(i+j)} \, b^{i+j}, $$ which is in $a^{m+n}K$, where the power is considered mod $4$ (since $a^4K = K$).
If this quotient group looks like the cyclic group of order $4$, that's because it is isomorphic! To make this explicit, say $C_4 = \langle c \rangle = \{e, c, c^2, c^3\}$, where $c^4=e$. Then the correspondence is \begin{array}{c@{50em}cc} G/K &\longleftrightarrow& C_4 \\ a^mK && c^m \end{array}
Putting all this together, the quotient group is $$ G/K \cong C_4. $$