As I said, you need to find all subgroups of each group in question up to conjugacy.
See this thread for reference. In my answer here, $\text{Id}$ denotes the trivial group, whereas $C_r$ denotes the cyclic group of order $r$ (well, I do not like to use $\mathbb{Z}_r$ to talk about the cyclic group). I also assume that $D_{n}$ is the dihedral group of order $2n$ (some people use this notation to mean the dihedral group of order $n$, which means something only when $n$ is even). As usual, $A_n$ is the alternating subgroup of the symmetric group $S_n$.
Subgroups of $S_4$ up to conjugacy are $\text{Id}$, two kinds of $C_2$, $C_3$, two kinds of $C_2\times C_2$, $C_4$, $S_3$, $A_4$, and $S_4$. Thus, the corresponding transitive $S_4$-sets are $S_4/\text{Id}\cong S_4$, $S_4/\langle (1\;2)\rangle$, $S_4/\langle (1\;2)(3\;4)\rangle$, $S_4/\langle(1\;2\;3)\rangle$, $S_4/\langle (1\;2),(3\;4)\rangle$, $S_4/\langle (1\;2)(3\;4),(1\;3)(2\;4)\rangle \cong S_3$, $S_4/\langle (1\;2\;3\;4)\rangle$, $S_4/\langle (1\;2),(1\;2\;3)\rangle=S_4/S_3$, $S_4/A_4\cong C_2$, and $S_4/S_4\cong \text{Id}$.
For $D_{10}$, the subgroups up to conjugacy are $\text{Id}$, three kinds of $C_2$, $D_2\cong C_2\times C_2$, $C_5$, $C_{10}$, two kinds of $D_{5}$, and $D_{10}$. Write $D_{10}:=\big\langle a,b\,\big|\,a^5=1\,,\,\,b^2=1\,,\text{ and }ba=a^{-1}b\big\rangle$. Thus, the corresponding transitive $D_{10}$-sets are $D_{10}/\text{Id}\cong D_{10}$, $D_{10}/\langle b\rangle$, $D_{10}/\langle ab\rangle$, $D_{10}/\langle a^5\rangle \cong D_5$, $D_{10}/\langle a^5,b\rangle$, $D_{10}/\langle a^2\rangle\cong D_2\cong C_2\times C_2$, $D_{10}/\langle a\rangle\cong D_1\cong C_2$, $D_{10}/\langle a^2,b\rangle$, $D_{10}/\langle a^2,ab\rangle$, and $D_{10}/D_{10}\cong \text{Id}$.
For $C_{10}:=\big\langle c\,\big|\,c^{10}=1\big\rangle$, it has only four subgroups: $\text{Id}$, $\langle c^5\rangle\cong C_2$, $\langle c^2\rangle\cong C_5$, and $C_{10}$. Thus, the corresponding transitive $C_{10}$-sets are simply $C_{10}/\text{Id}\cong C_{10}$, $C_{10}/\langle c^5\rangle\cong C_5$, $C_{10}/\langle c^2\rangle\cong C_2$, and $C_{10}/C_{10}\cong\text{Id}$.
Subgroups of $S_3\times C_4$ up to conjugacy are of the form $H\times N$, where $H$ is a subgroup of $S_3$ and $N$ is a subgroup of $C_4$, a diagonal version of $C_2$, and a diagonal version of $S_3$. The list of $H$ (again, up to conjugacy) is: $\text{Id}$, $\langle(1\;2)\rangle$, $\langle (1\;2\;3)\rangle\cong A_3\cong C_3$, and $S_3$. The list of $N$ is simply $\text{Id}$, $\langle g^2\rangle\cong C_2$, and $C_4$, where $C_4:=\big\langle g\,\big|\,g^4=1\big\rangle$, so that $C_4/\text{Id}\equiv C_4$, $C_4/\langle g^2\rangle\cong C_2$, and $C_4/C_4\cong \text{Id}$. Diagonal versions of $C_2$ and $S_3$ are simply the subgroups (up to conjugacy) $$\tilde{C}_2:=\Big\langle \big((1\;2),g^2\big)\Big\rangle\cong C_2\text{ and }\tilde{S}_3:=\Big\langle \big((1\;2),g^2\big),\big((1\;2\;3),1\big)\Big\rangle\cong S_3\,.$$ Hence, if $G:=S_3\times C_4$, then the transitive $G$-sets up to isomorphism are $S_3\times C_4=G$, $\big(S_3/\langle (1\;2)\rangle\big)\times C_4$, $\big(S_3/A_3\big)\times C_4\cong C_2\times C_4$, $(S_3/S_3)\times C_4\cong \text{Id}\times C_4\cong C_4$, $S_3\times C_2$, $\big(S_3/\langle (1\;2)\rangle\big)\times C_2$, $\big(S_3/A_3\big)\times C_2\cong C_2\times C_2$, $(S_3/S_3)\times C_2\cong \text{Id}\times C_2\cong C_2$, $S_3\times \text{Id}\cong S_3$, $\big(S_3/\langle (1\;2)\rangle\big)\times \text{Id}\cong \big(S_3/\langle (1\;2)\rangle\big)$, $\big(S_3/A_3\big)\times \text{Id}\cong C_2\times \text{Id}\cong C_2$, $(S_3/S_3)\times \text{Id}\cong \text{Id}\times \text{Id}\cong \text{Id}$, $G/\tilde{C}_2$, and $G/\tilde{S}_3\cong C_2$.
Best Answer
Indeed, if $G\curvearrowright\Omega$ transitively then the (general version of the) orbit-stabilizer theorem says $\Omega\cong G/S$ as a $G$-set, where $S$ is the stabilizer of any element in $\Omega$ (these stabilizers are conjugate so the corresponding coset spaces are equivalent). Which in turn means if $H\le G$ then the orbit space $H\backslash \Omega$ is in bijection with the double coset space $H\backslash G/S$, and there is in turn a surjection to the double coset space from the right coset space $H\backslash G$, which makes $[G:H]$ an upper bound. This is basically a different perspective on your own proof.
So you're right the first part of the theorem doesn't require the hypothesis that $H$ is normal. But the second part of the theorem does require this hypothesis. If $N$ is normal, then the quotient group $G/N$ acts on the orbit space $N\backslash\Omega$, transitively because $G\curvearrowright\Omega$ was transitive, which by orbit stabilizer implies the size of the orbit space divides the size of $G/N$. But if $H$ is not normal, the $H$-orbits can have different sizes, and there is no guaranteed well-defined projection from $G/H$ to $H\backslash\Omega$ with equal-size fibers. And indeed, we can cook up a counterexample by looking at small groups presumably.